modal
modal is the Python SDK and CLI for Modal, a hosted serverless compute platform. You decorate an ordinary Python function with @app.function(), describe the container it should run in with modal.Image, and calling my_function.remote(x) ships the arguments to Modal's cloud, starts a container there, runs the function, and returns the result to your laptop. The same decorator covers GPU jobs, cron schedules, HTTP endpoints, and fan-out across thousands of containers with .map(). There is no Dockerfile to write, no Kubernetes manifest, and no cluster to keep alive; the image is defined in Python and built on their infrastructure. The important thing to understand before you install it: this package is a client. It does nothing on its own. Every meaningful call goes over gRPC to modal.com, needs an account and a token, and gets billed per second of container time.
For bursty GPU work and fan-out batch jobs, Modal removes more infrastructure work than anything else with a Python API, and the developer experience is genuinely good. Go in clear-eyed that you are writing code against one vendor's platform with no exit path and a per-second bill, which is a strategic choice, not just a library choice.
Use it if
- You need GPUs occasionally and do not want to own them. Requesting gpu='A100' on a decorator and paying for the minutes you use beats renting an instance that idles overnight
- Your workload is embarrassingly parallel. .map() over ten thousand inputs spreads across containers with no queue, worker pool, or autoscaler for you to operate
- You want the container image defined in the same file as the code, with layer caching handled remotely: Image.debian_slim().apt_install('ffmpeg').pip_install('torch') replaces a Dockerfile plus a registry plus a build pipeline
- You are shipping an inference or batch-job endpoint and the interesting engineering is the model, not the infrastructure; @modal.fastapi_endpoint() gives you a URL from a Python function
- You need scheduled jobs and web endpoints from the same codebase without standing up a scheduler and a web service separately
- You need to run anywhere else. This is a client for one company's proprietary platform. There is no local runtime, no self-hosted server, and no compatible competitor to fail over to, so the decorators on your functions are a rewrite if Modal changes its pricing or shuts a product down
- Cost predictability matters more than convenience. Billing is per-second container time and you can leave containers warm with min_containers; a misconfigured scaledown_window or a runaway .map() bills real money with no local dry-run mode to catch it
- You are pinning dependencies in a machine learning environment. modal requires protobuf >=3.19,<7.0 excluding 4.24.0, grpclib >=0.4.7,<0.4.10, and click ~=8.1, and it pulls aiohttp, rich, watchfiles, cbor2, and synchronicity. In a project that already has grpcio, protobuf, or an older click, expect a resolver fight
- You want ordinary Python debugging. Your code runs in someone else's container, so pdb, local breakpoints, and profilers do not apply; you get remote logs, modal shell, and .local() for the parts that run on your machine. The sync and async dual API is generated by the synchronicity wrapper, which makes tracebacks and editor introspection less direct than plain asyncio
- You are on Python 3.9 or 3.15. The package raises a RuntimeError at import time outside the 3.10 to 3.14 range, which is a narrower window than most libraries enforce
- Cold starts are in your latency budget. A container that is not already warm has to be scheduled, pulled, and started before your function body runs, and keeping one warm to avoid that means paying for idle time
Setup reality
pip install modal, then run modal setup, which opens a browser to create an API token and writes it to ~/.modal.toml; in CI you set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET as environment variables instead. That much is smooth. The friction starts with the dependency set: aiohttp, grpclib, protobuf, rich, click, watchfiles, cbor2, toml, and synchronicity all land in your environment, and the pins on grpclib and protobuf are tight enough to conflict with anything else that speaks gRPC. Version 1.0 renamed a lot of things and the old names are gone or deprecated: modal.Stub is now modal.App and raises an AttributeError telling you so, @modal.web_endpoint is now @modal.fastapi_endpoint, keep_warm became min_containers, concurrency_limit became max_containers, container_idle_timeout became scaledown_window, and allow_concurrent_inputs was replaced by the @modal.concurrent decorator. Any tutorial written before mid-2025 will not run. The mental model also takes a while: code at module level runs both locally and inside the container, so imports of heavy libraries belong inside the function body or inside image.imports(), and files are not shipped unless you add them with add_local_dir or add_local_python_source. Finally, note the release cadence: PyPI carries over 2600 releases including a dev build most nights, so pin an exact version in production rather than tracking the latest.
Patterns
Run a Python function in the cloudfirst-remote-function
# app.py
import modal
app = modal.App("example")
@app.function()
def square(x: int) -> int:
return x * x
@app.local_entrypoint()
def main():
print(square.remote(4)) # runs on Modal
print(square.local(4)) # runs on your machine
# modal run app.pyModule-level code runs both locally and in the container, so keep heavy imports inside the function. .remote() blocks until the result comes back, .local() skips Modal entirely, and calling square(4) directly raises rather than guessing which you meant.
Describe the container image in Pythondefine-image
image = (
modal.Image.debian_slim(python_version="3.12")
.apt_install("ffmpeg", "git")
.pip_install("torch==2.5.1", "transformers")
.env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
.add_local_python_source("my_package")
)
@app.function(image=image)
def run():
import torch
return torch.__version__Each call adds a cached layer, so put slow steps first and volatile ones last. Your local files are not in the image unless you add them: add_local_python_source for modules, add_local_dir for data. There is also Image.from_registry(tag) for an existing Docker image, which must be linux/amd64.
Ask for a GPU and a longer timeoutgpu-function
@app.function(
image=image,
gpu="A100-40GB",
timeout=60 * 30,
retries=modal.Retries(max_retries=2, backoff_coefficient=2.0),
memory=16384,
)
def train(batch: list[str]) -> str:
import torch
assert torch.cuda.is_available()
return "done"timeout defaults to 300 seconds, which kills long training runs without much explanation. gpu also accepts a list such as ['H100', 'A100'] so the scheduler can fall back when your first choice has no capacity.
Fan out across many containersparallel-map
@app.local_entrypoint()
def main():
results = list(square.map(range(10_000)))
# arguments as tuples
list(add.starmap([(1, 2), (3, 4)]))
# fire and forget, ignore return values
square.for_each(range(1000))
# detach one call and poll for it later
call = square.spawn(7)
print(call.get(timeout=60))map() returns a lazy generator, so nothing runs until you iterate it. Container count is bounded by max_containers on the decorator; without one, a large map can start a very large number of containers and the bill scales with them.
Load a model once per container with a classclass-with-startup
@app.cls(image=image, gpu="L4", scaledown_window=300)
class Embedder:
model_name: str = modal.parameter(default="BAAI/bge-small-en")
@modal.enter()
def load(self):
from sentence_transformers import SentenceTransformer
self.model = SentenceTransformer(self.model_name)
@modal.method()
def embed(self, texts: list[str]):
return self.model.encode(texts).tolist()
@modal.exit()
def shutdown(self):
del self.model
@app.local_entrypoint()
def main():
print(Embedder().embed.remote(["hello"]))@modal.enter() runs once when a container starts, not once per call, which is the whole point for model loading. Class attributes must be declared with modal.parameter() to become constructor arguments; a plain __init__ is not how you pass configuration here.
Expose a function as a URLhttp-endpoint
web_image = modal.Image.debian_slim().pip_install("fastapi[standard]")
@app.function(image=web_image)
@modal.fastapi_endpoint(method="POST", docs=True)
def predict(item: dict):
return {"ok": True, "echo": item}
# modal serve app.py -> temporary URL that hot reloads
# modal deploy app.py -> permanent URLDecorator order matters: @app.function() goes outside, @modal.fastapi_endpoint() inside. This replaced @modal.web_endpoint, which is still importable but deprecated. fastapi is not in the default image, so add it or the deploy fails at build time.
Serve a whole FastAPI appasgi-app
from fastapi import FastAPI
web_app = FastAPI()
@web_app.get("/health")
def health():
return {"status": "ok"}
@app.function(image=web_image)
@modal.asgi_app()
def fastapi_app():
return web_appThe decorated function is a factory: it runs inside the container and returns the ASGI app, so build the app at module level or inside the function, but do not import server-only dependencies at the top of a file you also run locally.
Inject credentials without committing themsecrets
@app.function(
secrets=[
modal.Secret.from_name("openai-key", required_keys=["OPENAI_API_KEY"]),
modal.Secret.from_dict({"STAGE": "prod"}),
]
)
def call_api():
import os
return os.environ["OPENAI_API_KEY"][:6]
# modal secret create openai-key OPENAI_API_KEY=sk-...Secrets arrive as environment variables inside the container only. required_keys makes a missing key fail at deploy time instead of with a KeyError in the middle of a run. from_dict is for non-sensitive config; it is visible in your source.
Keep files between runs with a Volumepersistent-volume
vol = modal.Volume.from_name("model-cache", create_if_missing=True)
@app.function(image=image, volumes={"/cache": vol})
def download():
from pathlib import Path
Path("/cache/weights.bin").write_bytes(b"...")
vol.commit() # make it visible to other containers
@app.function(volumes={"/cache": vol})
def read():
vol.reload()
return open("/cache/weights.bin", "rb").read()Writes are not durable until commit(), and other containers do not see them until they reload(). reload() fails if the volume has open file handles. Volumes are tuned for a moderate number of large files, not for many small ones.
Run something on a schedulescheduled-job
@app.function(schedule=modal.Cron("0 9 * * *", timezone="Asia/Kolkata"))
def daily_report():
...
@app.function(schedule=modal.Period(hours=4))
def every_four_hours():
...
# modal deploy app.pySchedules only fire for a deployed app; modal run executes once and exits. Cron takes a timezone, Period does not and simply measures the interval from the previous run. A scheduled function cannot take arguments.
Invoke a deployed function from another processcall-deployed-code
import modal
f = modal.Function.from_name("example", "square")
print(f.remote(4))
Embedder = modal.Cls.from_name("example", "Embedder")
print(Embedder().embed.remote(["hello"]))
q = modal.Queue.from_name("jobs", create_if_missing=True)
q.put({"id": 1})This is how a normal web server or worker calls into Modal without defining the app locally. Lookups are lazy and resolve on first use, so a wrong app or function name surfaces as a NotFoundError at call time rather than at import.
Control input concurrency and container countsconcurrency-and-scaling
@app.function(
min_containers=1, # keep one warm, you pay for it
max_containers=50, # hard ceiling on fan-out
buffer_containers=2, # spin up ahead of demand
scaledown_window=120, # seconds idle before shutdown
)
@modal.concurrent(max_inputs=10, target_inputs=6)
async def handle(req: dict):
...@modal.concurrent replaced the allow_concurrent_inputs parameter and only helps for IO-bound work; running ten CPU-bound inputs in one container just makes each slower. min_containers is the old keep_warm and is the fastest way to run up a bill you did not expect.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| runpod | PyPI | You mainly want cheap on-demand GPU containers and are happy managing your own Docker image and handler contract |
| ray | PyPI | You want the same remote-function and fan-out model but running on infrastructure you control, on-prem or in your own cloud account |
| bentoml | PyPI | You are packaging and serving models and want an open-source framework that deploys to Kubernetes or any cloud rather than one vendor |
| prefect | PyPI | The actual need is scheduled and retried batch workflows rather than on-demand GPU compute |