PyPIDataupdated 06 Aug 2026

databricks-sql-connector

Item: databricks-sql-connector
Rating: 3.8
Author: Keyoor

The Databricks SQL Connector is a pure Python, DB API 2.0 client for Databricks SQL warehouses and all-purpose clusters. It speaks Thrift over HTTPS, so there is no ODBC or JDBC driver to install on every machine, and it moves result sets as Apache Arrow batches instead of row-by-row structs. You open a connection with a workspace hostname, an HTTP path pointing at a warehouse, and either a personal access token or an OAuth credentials provider, then use the same cursor.execute and fetchall shape you already know from sqlite3 or psycopg. Everything else (Unity Catalog metadata lookups, volume file ingestion, query cancellation) hangs off that cursor.

databricks-sql-connector · v4.4.0 · 27.9M/wk · ★ 233 · pushed 05 Aug 2026 · databricks/databricks-sql-python · docs · reviewed by Keyoor

Verdict

The official and actively maintained way to run SQL against Databricks from Python, and the only one that avoids an ODBC install. Budget for a heavy dependency tree and accept that the real reference documentation lives on docs.databricks.com, not in the repo.

API stability4/5DB API 2.0 pins the core surface, and the native named-parameter change in 3.0.0 shipped with a compatibility path that rewrites old pyformat markers for you. Major versions still cut things: 4.0.0 moved the SQLAlchemy dialect out into its own package.

Docs3/5The examples directory is genuinely useful (parameters, OAuth U2M and M2M, transactions, staging ingestion, cancellation), but the README is a quickstart only and the reference lives on the Databricks docs site. The README also states a Python 3.9 minimum that the 4.4.0 package metadata contradicts.

Maintenance5/5Databricks staffs this directly, main was pushed the day before this review, and 4.4.0 is the current release. 83 open issues against a vendor driver of this size is a normal backlog, not neglect.

Ecosystem3/5Everything Databricks-flavoured sits on it (dbt-databricks, databricks-sqlalchemy, notebook tooling), but it is a single-vendor driver. Nothing you write against it ports to another warehouse without a rewrite of the connection layer.

Use it if

You need to query a Databricks SQL warehouse or cluster from Python and do not want to install and version an ODBC or JDBC driver on every host and container
Your code already speaks DB API 2.0 and you want the same cursor.execute / fetchall / description surface pointed at Databricks
You want results as Arrow (fetchall_arrow, fetchmany_arrow) so you can hand them to pandas or Polars without a per-row Python conversion
You need Databricks OAuth, either the U2M browser flow for local work or M2M service principals for jobs, instead of long-lived personal access tokens
You want to drive Unity Catalog volume ingestion (PUT, GET, REMOVE of local files) from the same connection that runs your queries

Skip it if

You care about install size. pandas, openpyxl, thrift, lz4, pybreaker, pyjwt, oauthlib, python-dateutil and requests are all hard dependencies, so a container that runs one SELECT still pulls the full pandas stack and an Excel writer.
You want SQLAlchemy. The dialect was removed from this package in 4.0.0 and now lives in databricks-sqlalchemy, a separate release with its own version skew against the connector core.
You need low-latency point lookups. Every statement is a network round trip to a warehouse that may be asleep, and cold starts measured in seconds to minutes are normal. No client-side tuning fixes that; put an OLTP database in front instead.
You plan to insert rows one at a time. executemany here is documented as issuing N sequential requests with no batching, so a 10,000-row load becomes 10,000 queries.
You want a community-sized support surface. The repo has 233 stars and 83 open issues (152 issues and PRs combined), and the real answer to most questions is a Databricks support ticket.
You are still on Python 3.9. The README says 3.9 or above, but the 4.4.0 wheel metadata requires >=3.10,<4.0, and the metadata is what pip enforces.

Setup reality

pip install databricks-sql-connector gives you a working client, but the Arrow fetch paths need the extra: pip install 'databricks-sql-connector[pyarrow]'. Then you need three values that people routinely mix up. server_hostname is the bare host with no https:// prefix and no trailing slash. http_path is copied from the warehouse Connection Details tab and looks like /sql/1.0/warehouses/abc123, not your workspace URL. The credential is either access_token (a PAT), or credentials_provider, which for OAuth M2M requires installing databricks-sdk separately and assigning the service principal to the workspace, a step that is easy to forget and produces an unhelpful 403. The [kernel] extra that switches on the Rust backend installs nothing at all on Python 3.9, and use_kernel=True then raises ImportError rather than falling back.

Patterns

Connect with a token and run a queryconnect-and-query

import os
from databricks import sql

with sql.connect(
    server_hostname=os.getenv("DATABRICKS_SERVER_HOSTNAME"),
    http_path=os.getenv("DATABRICKS_HTTP_PATH"),
    access_token=os.getenv("DATABRICKS_TOKEN"),
) as connection:
    with connection.cursor() as cursor:
        cursor.execute("SELECT * FROM range(10)")
        for row in cursor.fetchall():
            print(row)

server_hostname is the bare host with no scheme and no trailing slash. Both the connection and the cursor are context managers; skipping them leaks a server-side session until garbage collection.

Bind parameters server-sidenamed-parameters

with connection.cursor() as cursor:
    cursor.execute(
        "SELECT :name AS name, :age AS age, :active AS active",
        {"name": "Jane", "age": 30, "active": True},
    )
    print(cursor.fetchone())

Since 3.0.0 parameters are bound by the server by default. Legacy pyformat markers like %(name)s still work and get rewritten to :name for you. Pass use_inline_params="silent" to force the old client-side string substitution, which is the only mode that is vulnerable to injection.

Turn rows into dictsread-rows-as-dicts

cursor.execute("SELECT id, name FROM main.default.users LIMIT 5")
rows = cursor.fetchall()

print(rows[0][0])          # positional, Row subclasses tuple
print(rows[0].name)        # attribute access
print(rows[0].asDict())    # {"id": 1, "name": "Jane"}

Row is a tuple subclass, so positional indexing, attribute access and asDict() all work on the same object. cursor.description holds the column metadata if you need types.

Fetch results as an Arrow tablefetch-arrow-table

cursor.execute("SELECT * FROM samples.nyctaxi.trips LIMIT 100000")
table = cursor.fetchall_arrow()   # pyarrow.Table
df = table.to_pandas()

# or stream it in chunks
cursor.execute("SELECT * FROM big_table")
while (batch := cursor.fetchmany_arrow(50_000)).num_rows:
    process(batch)

Both methods need the extra: pip install 'databricks-sql-connector[pyarrow]'. This is the fast path; fetchall() converts every cell into a Python object and is much slower on wide result sets.

Insert many rows without one query per rowbulk-insert

with connection.cursor() as cursor:
    cursor.execute("CREATE TABLE IF NOT EXISTS squares (x int, x_squared int)")

    squares = [(i, i * i) for i in range(1000)]
    values = ",".join(f"({x}, {y})" for x, y in squares)
    cursor.execute(f"INSERT INTO squares VALUES {values}")

executemany is documented as running the statement once per parameter set with no batching, so it is the wrong tool for loading data. For anything past a few thousand rows, write files to a volume and run COPY INTO instead of building a giant SQL string.

Start a long query without blockingasync-execute

import time

with connection.cursor() as cursor:
    cursor.execute_async("SELECT COUNT(*) FROM huge_fact_table")

    while cursor.is_query_pending():
        time.sleep(5)

    cursor.get_async_execution_result()
    print(cursor.fetchall())

This is polling, not asyncio. The thread still sleeps between checks, so it buys you the ability to survive a client restart or do other work, not concurrency for free.

Cancel a running statementcancel-query

import threading
from databricks import sql

def run():
    try:
        cursor.execute("SELECT SUM(a.id - b.id) FROM range(1000000000) a CROSS JOIN range(100000000) b GROUP BY (a.id - b.id)")
    except sql.exc.RequestError:
        print("cancelled")

threading.Thread(target=run).start()
time.sleep(15)
cursor.cancel()

cancel() must be called from another thread because execute() blocks. The cursor stays usable afterwards, so you can run a new statement on it without reconnecting.

Group statements in a transactiontransactions

connection.autocommit = False

with connection.cursor() as cursor:
    try:
        cursor.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1")
        cursor.execute("UPDATE accounts SET balance = balance + 100 WHERE id = 2")
        connection.commit()
    except Exception:
        connection.rollback()
        raise
    finally:
        connection.autocommit = True

autocommit defaults to True, so without setting it False every statement commits on its own. Multi-statement transactions need a Databricks runtime that supports them; older warehouses will reject the mode.

Authenticate as a service principaloauth-m2m

import os
from databricks.sdk.core import Config, oauth_service_principal
from databricks import sql

hostname = os.getenv("DATABRICKS_SERVER_HOSTNAME")

def credential_provider():
    return oauth_service_principal(Config(
        host=f"https://{hostname}",
        client_id=os.getenv("DATABRICKS_CLIENT_ID"),
        client_secret=os.getenv("DATABRICKS_CLIENT_SECRET"),
    ))

connection = sql.connect(
    server_hostname=hostname,
    http_path=os.getenv("DATABRICKS_HTTP_PATH"),
    credentials_provider=credential_provider,
)

databricks-sdk is not a dependency of this package, so pip install databricks-sdk yourself. The service principal must be explicitly assigned to the workspace, otherwise you get a 403 that says nothing about the missing assignment.

List catalogs, schemas, tables and columnslist-catalog-metadata

with connection.cursor() as cursor:
    print(cursor.catalogs().fetchall())
    print(cursor.schemas(catalog_name="main").fetchall())
    print(cursor.tables(catalog_name="main", schema_name="default").fetchall())
    print(cursor.columns(catalog_name="main", schema_name="default", table_name="users").fetchall())

Each method returns the cursor itself, so you chain .fetchall() on the end. Name arguments accept % as a wildcard, which is how you do prefix searches without writing SHOW statements.

Upload a local file to a staging volumevolume-ingestion

import os
from databricks import sql

local_dir = os.path.dirname(os.path.realpath("september.csv"))

with sql.connect(
    server_hostname=os.getenv("DATABRICKS_SERVER_HOSTNAME"),
    http_path=os.getenv("DATABRICKS_HTTP_PATH"),
    access_token=os.getenv("DATABRICKS_TOKEN"),
    staging_allowed_local_path=local_dir,
) as connection:
    with connection.cursor() as cursor:
        cursor.execute(
            "PUT 'september.csv' INTO 'stage://tmp/me@example.com/sales/september.csv' OVERWRITE"
        )

Without staging_allowed_local_path the PUT is refused, and the connector will only touch files inside that directory tree. GET and REMOVE use the same connection. You can only write under the authenticated user's own staging location.

Opt into the Rust kernel backenduse-kernel-backend

# pip install "databricks-sql-connector[kernel]"
connection = sql.connect(
    server_hostname=host,
    http_path=http_path,
    access_token=token,
    use_kernel=True,
)

The kernel wheel is published as cp310-abi3, so on Python 3.9 the extra installs nothing and use_kernel=True raises ImportError instead of falling back to Thrift. The extra also drags in PyArrow, which the kernel result path requires.

Alternatives

Package	Registry	Pick it when
databricks-sdk	PyPI	You need workspace, jobs, clusters or Unity Catalog management APIs rather than SQL execution
databricks-sqlalchemy	PyPI	You want SQLAlchemy Core or an ORM on top of Databricks; the dialect moved here in 4.0.0
pyodbc	PyPI	You already deploy the Databricks ODBC driver everywhere and want one code path across several warehouses