Design and limitations¶

Cereyan is built for one machine, one wheel, and pipelines that can always be rerun. Those three choices explain most of what it does and everything it refuses to do. This page says what those choices mean for you.

What it is for¶

A data engineer with pipelines on a laptop, a workstation, or one server, who wants schedules, retries, backfills, a UI, and alerts without running a platform.
Batch work measured in seconds to hours, tens of thousands of runs a day at most, where files and tables written by the flow are the real output.
Teams that keep flows in one repository and prefer code over configuration.

Decisions¶

One wheel, no runtime dependencies. pip install cereyan is the whole install: the Rust core, the server, the UI, and the MCP server are inside the wheel. Nothing is pulled in at run time, so a pipeline environment gains no transitive dependencies from its orchestrator. Docs tooling and tests use dependency groups that never reach the wheel.

One process serves everything. cereyan serve is the API, the UI, the scheduler, the rules engine, the MCP endpoint, the one queue, and the engine supervisor. There is no broker or separate queue to run. Other machines can add processors with cereyan worker, which connects out to the server, runs engines from its own checkout, and keeps nothing of its own: the store, results, secrets and the flow definitions stay on the server. See Run across machines.

Remote workers, decided for 3.0. Earlier releases listed remote workers as out of scope, because distributing execution seemed to need a broker, a scheduler that knows about hosts, and a rollout story. By 3.0 the server already was the broker: engines pull their next run over HTTP with a token, and the queue picks the first run that can start. A worker is therefore an engine on another machine plus a registration: it declares which flows its checkout can run and a fingerprint per module, and a run only goes where the code matches. Work pools, per-queue routing, and shipping code to workers remain out of scope.

Two execution paths, one store. A plain python pipeline.py writes the same SQLite store the server uses, under the same state rules, so a script and a served run look identical in history. When a server holds the store, scripts hand their runs to it.

History is a cache. The database records what happened; your code and your targets are the source of truth. Losing db.sqlite loses the run list, not your data, and a corrupted file is quarantined rather than blocking startup. Crashed runs are rerun, up to a limit, on the assumption that a rerun is safe because targets make it so.

A fixed vocabulary. App, flow, task, run, task run, state, schedule, parameter, target, resource, backfill, artifact, variable, event, rule. There is no "deployment": a served flow with a schedule is the deployed thing.

Performance is a feature. The store is SQLite in WAL mode with one writer, group commit, a read pool, and an in-memory working set; engines report in batches serialised in Rust. The targets are enforced by benchmarks in CI.

What it does not do, and why¶

Not provided	Reason	Instead
Work pools, per-queue routing, Kubernetes or Docker execution, shipping code to workers	One queue serves every host, and a worker runs its own checkout; building images, choosing infrastructure per run, or copying code to machines is the platform Cereyan set out not to be.	`cereyan worker` on each machine with a checkout; `runs_on="server"` for flows that must stay on the server; call remote systems from inside tasks.
Postgres or any other database	SQLite gives one file, no service, and the performance the targets need; a second backend would double the store and change the operational shape.	Retention keeps the file bounded; back it up like any file.
Remote or object-store targets (S3, GCS, HDFS)	A `Target` is anything with `exists()`, so you can write one in a few lines with the client library you already use; shipping them would add dependencies to the wheel.	Write a small class with `exists()`.
Integrations and connector packages	Tasks are plain Python; the library for your warehouse or API works unchanged inside one.	Import it in the task.
Multi-user accounts, roles, SSO, audit trails	One token, one permission level, one machine.	Put a reverse proxy in front for network access; use the token or the socket.
Cloud-style extras: assets, SLAs, incident management, metric triggers, incoming webhooks as event sources	Each is a product on its own; rules with `unless` and custom routes cover the local versions of the common cases.	A proactive rule for "did not happen by"; `fresh_within`, `expect_by` and `expected_duration` on a flow as shorthand, shown as PASS, WARN or FAIL; a custom route that calls `emit_event` for incoming webhooks.
TLS	The server is loopback-only by default and a reverse proxy does TLS better.	nginx (Secure the server has a configuration), Caddy, or an SSH tunnel.
Versioned flows, code storage, image builds	The code in the served directory is the version; engines reload it when it changes.	Git.
Real-time or streaming pipelines	Runs are units of work with a beginning and an end; the scheduler is a timer heap, not an event loop over streams.	A long-running flow per stream partition, or another tool.
Preemption, priority-based killing	A running run is never interrupted to make room for a higher-priority one; priority only orders the queue. The one exception is a flow that declares `on_overlap="cancel_old"`, which asks its own older runs to stop.	Resources and caps to keep heavy work from starting; `cancel_old` for latest-wins flows.

Limits worth knowing¶

One server per machine, because the home and its advisory lock are global. A machine can instead run a worker for another machine's server.
Workers are on the same network as the server, or reach it through a TLS proxy; the server itself does not terminate TLS. All workers share the server's token.
A worker's files are its own: a LocalTarget it writes outside its declared shared paths is on that machine only, and is reported as run.local_path_on_worker.
The engine pool bounds concurrent runs at max_engines (processors on the Queue page), default 1 and at most the CPU count; a run occupies an engine for its whole duration, including time spent waiting on I/O, unless it waits through sleep, wait_for_event, wait_for_target, or wait_for_input, which free the engine.
Artifacts are limited to 1 MB and variables to 64 KB.
Retention deletes logs and events older than retain_days, and runs older than retain_runs_days when that is set; persisted results under storage/ are a cache keyed by task inputs and are not deleted per run.
Unix sockets and engine niceness do not exist on Windows.
A timeout or a cancel interrupts a running flow in Python code and in time.sleep on every platform, and in any blocking system call on Linux and macOS. A flow stuck in a call the platform cannot interrupt, such as a loop inside a C extension or most blocking I/O on Windows, is timed out when the call returns offline, and ended by the supervisor when served.
Schedules, dependencies, backfills, data rules, clock-armed proactive rules, and pausing need a running server; offline scripts record runs and fire code rules that a server has registered.