v3.3.1 MIT licensed Python 3.11+ Rust core

Orchestration that starts as one local file.

Cereyan is a minimal, local-first orchestrator for Python data pipelines. A Rust core sits behind a thin layer of decorators, and the whole thing ships as one wheel with no runtime dependencies.

Read the quickstart Take the tour

$ pip install cereyan

pipeline.py python

from datetime import date
from cereyan import flow, task

@task
def extract(day: date) -> list[int]:
    return [1, 2, 3]

@flow(run_name="etl-{day}")
def etl(day: date) -> int:
    return sum(extract(day))

assert etl(date(2026, 9, 6)) == 6   # recorded as a run in ~/.cereyan/db.sqlite

Four ways to run the same file.

No daemon to register with, no manifest to publish. The script is the pipeline.

$ python pipeline.py: records a run in ~/.cereyan/db.sqlite
$ cereyan run pipeline.py:etl --param day=2026-01-02: the same, with parameters and a summary
$ cereyan serve .: API, UI, scheduler and engines on 127.0.0.1:4200
$ cereyan worker . --host https://cereyan.internal: on another machine: its processors join the same queue

One process serves the whole thing.

The API, the web UI, the scheduler, a warm pool of engine processes, rules, and a built-in MCP server for agents, all behind a single cereyan serve.

Take the full tour

The cereyan dashboard beside the grouped sidebar: counts by state over a histogram, the Flows card with each flow's last ten runs and a failed run's logs open, then Upcoming, Needs attention with a paused and a failed run, and Running now

Offline first

A script records runs into a local SQLite file. Nothing else needs to be running, and nothing leaves the machine.

One process to serve

API, UI, scheduler, a queue of processors you size while it runs, rules and MCP in one command. One wheel, with no runtime dependencies to resolve.

Pipeline semantics

Targets, backfills, keyed resources, chaining and fan-in, unique keys, and rules that react to events or to their absence.

Durable runs

Every finished task is checkpointed, so a retry resumes after the last one. A run can sleep, wait for an event, or wait for a message without holding an engine.

Built for the things data pipelines actually do.

Targets

Reruns only do the missing work.

Declare where a task writes. When that output already exists the task ends Skipped and the body never runs, so rerunning a day, a failed run or an entire backfill is safe.

Idempotent reruns

@task(output=lambda day: LocalTarget(f"data/{day}.csv"))
def export(day: date) -> None:
    with LocalTarget(f"data/{day}.csv").open("w") as fh:
        fh.write(rows)

Skipped output exists · task_run.skipped

Events and rules

React to what happened, and to what did not.

Every transition is an event on a live feed. Rules match them and act, and a rule can also fire when an expected event never arrives, which is how a missed delivery becomes a page rather than silence.

Rules and events

Live events streaming

09:15:02Completedrun.completedetl-2026-01-02
09:14:58Failedtask_run.failedload, retry 2 of 3
09:14:41Overduerun.overduevendor_drop never arrived
09:14:12Pausedrun.pausedawaiting approval

Backfills

Cover a date range in one command.

A backfill creates a run per date or per listed value, respects the flow's resources, and pairs with targets so a second pass touches only the days that are still missing.

Backfill a date range

cereyan backfill daily --from 2026-01-01 --to 2026-01-28 28 runs

Completed Skipped Running Failed Scheduled

Targets the benchmarks enforce.

These numbers shaped the design, and CI fails when one is missed or regresses more than twenty percent against the checked-in baseline. Measured on a developer laptop; a small server does better.

How they are checked

Performance targets
Server start with 1M historical runs	under 200 ms
Task transitions ingested	20k per second
Log lines ingested	100k per second
Runs list query at 1M runs	under 10 ms
Schedule wake-up drift	under 50 ms
Backfill create of 10k runs	under 1 s
Warm-pool overhead, Scheduled to user code	under 5 ms
Counts endpoint	under 5 ms
Orchestration cost per completed task run	under 200 µs

Agents read the same store you do.

A built-in MCP server exposes runs, logs, states and actions as tools, so an assistant can diagnose a failure without a scraping layer. The whole site is published as one file for retrieval too.

llms.txt llms-full.txt MCP reference

Coming from Prefect or Luigi?

The decorators will look familiar, and the migration guide maps the concepts you already have onto flows, tasks, targets and rules, including the ones that do not translate cleanly.

Read the migration guide

Install it, run your script, and look at what happened.

Nothing to sign up for and nothing to configure. The first run is recorded before you have decided whether you like it.

$ pip install cereyan Read the quickstart