mrkeyoor.com_
Thu 29 Oct 20:48 UTC
Open Source3 min read

MiniLake runs Databricks API tests locally, with limits on what they prove

The open-source emulator executes SQL and Spark jobs locally. Its documented permission and SQL-dialect limits determine which tests still need a real workspace.

MiniLake, an open-source Databricks API emulator, received its 1.9.0 release on October 25. It gives developers a local target for Databricks client code without a cloud account. The boundary matters: its documentation says permissions are not enforced, so successful local tests cannot establish that access controls work in a real workspace.

The release notes identify a SQL fix that preserves literals when rewriting backtick identifiers, alongside dependency updates addressing Dependabot alerts. Those are the documented changes in 1.9.0. The broader capabilities below come from the project's current documentation, rather than a claim that this release introduced them. We reviewed those sources without installing MiniLake or running its tests.

What executes behind the API

According to the project's documentation, SQL statements execute through DuckDB. External Delta tables use actual Parquet files and a Delta transaction log: writes go through generated Spark jobs, while reads use DuckDB's Delta reader. Spark jobs run in sibling containers, with a subprocess fallback also documented in the repository.

That makes the execution path worth examining before choosing a test. A Databricks-shaped request can produce stored data and query results locally. Because the SQL engine is DuckDB, local query success should not be treated as proof of Databricks SQL compatibility.

The Spark documentation draws another consequential distinction: managed tables are DuckDB tables without files and are invisible to Spark. The project directs developers to external Delta tables when Spark needs to read the data. A test that only queries through the SQL endpoint would miss that boundary.

Where a passing test stops helping

The repository's known gaps say MiniLake accepts any token and applies an allow-all permission model. Identity records can be created, but they do not provide a way to sign in. Cluster lifecycle operations are also simulated state transitions; Spark compute happens through Jobs instead.

For a developer planning continuous integration, those limits suggest a practical division: use local runs to exercise supported client calls and data operations, then retain real-workspace checks for authorization and production SQL behavior. That is an inference from the documented architecture, not a compatibility result we measured. The repository also acknowledges that several central modules have mostly happy-path test coverage.

The useful thing to watch is whether future releases document more behavioral conformance, especially around the gaps the project already names. Until then, a green local run should be read against those limits before it becomes a reason to skip a workspace test.

Sources

  1. MiniLake v1.9.0 release notes
  2. MiniLake Spark and Delta Lake documentation
  3. MiniLake repository: supported services and known gaps