Resources and concurrency¶

from cereyan import flow, task

@task(resources={"db": 1})
def load(rows: int) -> int:
    return rows

@flow(max_concurrent=1, on_overlap="skip", priority=5, resources={"gpu": 1})
def nightly(rows: int = 10) -> int:
    return load(rows)

assert nightly() == 10
assert nightly.options["max_concurrent"] == 1

A resource is a named counting semaphore with a total set in cereyan.toml ([resources]) or on the Settings page. Flows and tasks declare how much of a resource each run holds; a run whose resources are not available waits as AwaitingResource and is dispatched as soon as they are. Resources are released on every terminal state, on pause, and when an engine dies, so a crash cannot leak capacity. They are shared by every project on the machine.

Offline, a resource is a local semaphore of size one, which serialises tasks holding the same name inside a single process.

A name can be a template over the run's parameters, api:{{ tenant }}, rendered when the run is admitted or the task starts; a [resources] key with * gives every matching name its own total, and tag:<name> totals limit runs by tag. See Keyed resources.

Per-flow cap and overlap¶

max_concurrent=N is a resource named after the flow with total N: at most N runs of the flow are Pending or Running at once. The default is unlimited. on_overlap decides what a new run does when the cap is reached:

Value	Behaviour
`enqueue` (default)	Wait as `AwaitingResource` and start when a slot frees
`skip`	End `Skipped` at once with the message `previous run still active` and a `run.skipped` event
`cancel_new`	End `Cancelled` with the same message

Priority¶

priority orders dispatch among runs waiting for an engine or a resource, higher first. It never preempts: a running run is never paused or killed to make room. A negative priority also raises the engine's OS niceness (min(19, -priority)) on Linux and macOS, so heavy background flows yield CPU to interactive work; such engines are pooled by their niceness and count toward max_engines.

Disable windows¶

disable_after=(count, window_seconds, persist_seconds) pauses every schedule of the flow for persist_seconds once it has failed count times within window_seconds, records flow.disabled, and resumes automatically with flow.enabled. Use it to stop a broken nightly flow from filling the run list until someone looks. The window survives a restart of the server: one still open when the server starts again ends on time, and one that ended while it was down resumes as it starts. Fires inside the window are never caught up; fires after it follow the schedule's catchup policy.

With a window of None, disable_after=(3, None, 6 * 3600) counts failures in a row instead: a Completed run resets the count to zero. That is the natural stop for a continuous schedule, whose loop is paused, holds no waiting run, and is seeded again when the pause ends.

Engines¶

The number of runs executing at once is also bounded by the engine pool: max_engines engine processes, called processors in the UI (default: 1, at most the CPU count, changeable on the Queue page while the server runs), each running one run at a time. A free processor takes the first run in line that can start; runs blocked by a resource or a cap are passed by the ones that can. Resources and caps decide which runs may proceed; the pool decides how many. See Engines and the home directory.

Backfills¶

A backfill has its own resource, sized by its concurrency, so a large date range never floods the pool; see Backfills.