Four language APIs sit over five processing toolsets
Spark's README lists 4 language APIs: Scala, Java, Python, and R, with R now marked deprecated. Above the execution engine sit Spark SQL and DataFrames, the pandas API on Spark, MLlib, GraphX, and Structured Streaming. That range lets one cluster handle interactive queries, scheduled transformations, stream processing, and some machine-learning or graph work. PySpark is especially useful when a Python team needs the JVM engine and connectors without rewriting application logic in Scala.
Breadth has a cost. A SQL query, a Python user-defined function, and a stateful streaming job can all use Spark, but they fail and scale for different reasons. Teams still need to understand partitions, shuffles, executors, driver memory, serialization, checkpoints, and the storage systems beneath the job. Spark reduces the number of engines a data group must standardize; it does not turn distributed execution into ordinary local dataframe code.
The 198.8 MB checkout demands platform-level setup
Our commit 2a7cfea checkout contained 25,789 files and about 704,921 source lines in 198.8 MB. GitHub identifies Scala as the primary repository language, while users also meet Java, Python, R, build scripts, web assets, and cluster-specific modules. New contributors should choose one module and one supported build profile before opening an IDE. Indexing or compiling everything creates a slow feedback loop and teaches little about the path they intend to change.
The README's documented source build uses Apache Maven and requires Java 17 or newer. Users of a prebuilt package can skip that build, then start a Scala shell with spark-shell or a Python shell with pyspark. Example programs can run with a local master, a standalone Spark URL, or YARN, and the repository also documents Kubernetes elsewhere. That flexibility is useful after the deployment target is chosen; it is confusing when a team treats every master as interchangeable.
What happened when we ran it
Our sandbox's Python path installed 35 packages in 30 seconds and consumed 37 MB. The detected build completed in 13 seconds, and pip-audit found 0 known vulnerabilities in those installed Python dependencies. We used commit 2a7cfea in an unprivileged Debian container with 3 CPUs and 8 GB of RAM. These figures describe the harness path it found, not the cost of compiling Spark's Scala and Java modules or provisioning a working cluster.
The harness found no test script or target, so it skipped tests and reported no test count. The README documents a different project workflow: build Spark first, then run ./dev/run-tests, with module-level guidance and a separate Kubernetes integration suite. We will not convert an absent generic target into a passing result. Before changing Spark itself, follow the repository's selected-module test instructions and reserve the full suite for a machine sized for that job.
A 13-second detected build does not cover Maven or a cluster
The 13-second result came from our detected Python ecosystem path. A full Spark source build follows ./build/mvn -DskipTests clean package, according to the README, and cluster behavior adds remote storage, a master, workers, networking, and credentials. A local shell is useful for checking transformations and APIs. It cannot expose executor loss, uneven partitions, object-store throttling, Kubernetes pod settings, or the difference between driver-local and executor-visible files.
Issue 58310 gives a concrete Kubernetes example. The report says --files content uploaded to object storage is localized to a driver path, so user code loses the original remote URI. SparkFiles.get() can locate a copy inside a task, while a path resolved on the driver does not identify the executor's copy. The proposed workaround reads through the driver and parallelizes the content, which the issue itself says does not scale. Test the exact submission and file-access path you plan to use.
Java 17 and Hadoop matching can decide whether a build works
Java 17 or newer is the README's current baseline, and Spark says a source build must match the Hadoop version used by the target cluster. Hadoop protocols and distribution-specific Hive components can differ, so a binary assembled for one environment may be the wrong artifact for another. Record the Spark branch, Java runtime, Scala binary line, Hadoop profile, storage connectors, and cluster manager together. Treat that matrix as part of the application release rather than as ambient infrastructure.
Data correctness also deserves configuration-specific tests. Open issue 58495 reports that the row-based Parquet reader can accept two incompatible primitive conversions that the vectorized reader rejects. In the examples, switching the vectorized-reader setting changes a clean failure into interpreted output. The issue was opened September 3, 2026 and proposes consistent rejection. Pipelines with evolving Parquet schemas should test both configured reader behavior and the exact type conversions they expect before an optimization flag reaches production.
September 2026 work is active despite an empty release endpoint
The last push was September 5, 2026, and open pull requests were updated repeatedly that day. GitHub reported 43,949 stars and 505 combined open issues and PRs. Recent work covered SQL planning, Kubernetes, PySpark timestamp types, metrics queries, and dependency changes. That activity and the README's large branch-specific CI table are better health evidence than the GitHub latest-release endpoint, which returned no published release for this repository. Apache distributes official Spark versions outside that feed.
Spark is an established engine with a current codebase and a very large operating surface. Our 30-second Python install makes PySpark approachable for a local experiment, while the skipped tests prevent any claim about the full checkout. Adopt Spark because data volume, shared cluster execution, or streaming semantics require it. If one process can answer the query, the cluster scheduler, Hadoop compatibility matrix, and 25,789-file source tree are costs with no matching payoff.