How to detect when something did not happen¶

A reactive rule fires when an event arrives. A proactive rule fires when an expected event does not arrive: the nightly load that never finished, the run that has been going for two hours, the daily file that was not produced by nine. Add unless to a rule and arm it from an event or from the clock.

Event-armed: a deadline after something starts¶

from cereyan import App

app = App("loads")

@app.rule(on="run.running", flow="etl", unless="run.completed", within=2 * 3600)
def etl_overran(event, run):
    print(f"{run['name']} did not complete within two hours")

spec = app.rules[0].spec()
assert spec["unless"]["events"] == ["run.completed"] and spec["within"] == 7200.0

on arms an expectation when the run starts, unless names the event that disarms it, and within is the deadline in seconds. The expectation is keyed by the run (or by the flow, for events without a run). If the run completes in time the expectation is met; if it fails, or is still running at the deadline, the rule fires. Expectations are stored, so one whose deadline passed while the server was down fires once on start.

Clock-armed: something should have happened by now¶

from cereyan import App

app = App("loads2")

@app.rule(at="0 9 * * *", tz="Europe/Istanbul", flow="daily_load", unless="run.completed")
def daily_load_missing(event, run):
    print("no daily_load completed before nine")

spec = app.rules[0].spec()
assert spec["at"] == {"cron": "0 9 * * *", "tz": "Europe/Istanbul"}

At each tick of at the rule fires unless a matching event occurred in the look-back window: within seconds when given, otherwise the time since the previous tick. Ticks missed while the server was down are skipped with a log line. Clock-armed rules need a running server. A rule does not fire until it has been watching for a whole window: until then its look-back would reach past its own creation, where nothing had happened yet, and the first tick after you save a rule would page you about it.

Freshness and deadlines in one line¶

The common cases need no rule at all. Three flow options derive a health of PASS, WARN or FAIL from the runs the server already records, shown in the Health column of the Flows page and in every flow summary of the API:

from datetime import timedelta
from cereyan import flow

@flow(fresh_within=timedelta(hours=26), expect_by="0 9 * * *", expect_by_tz="Europe/Istanbul", expected_duration=600)
def nightly_load() -> None:
    ...

fresh_within fails the flow when its last completed run is older than the window and warns past three quarters of it. expect_by names a cron by which a run must have completed: the flow fails when the latest deadline passed with no completed run since the previous one, and warns while a run is still active at the deadline. expected_duration (or overdue_factor=2.0, against the median of the last 20 completed runs) warns while a run has outlasted its expectation and records run.overdue once for that run, so @app.rule(on="run.overdue") pages exactly as a rule on run.failed would. None of this is a state machine of its own: the health is computed from runs when asked.

What the rule sees¶

A lapse is recorded as an expectation.lapsed event with resource rule/<id>, the related run and flow, and a payload carrying rule, flow, project, run, run_name, expected, deadline, and armed_at. The rule's actions execute against that event, so templates can say:

{{ event.payload.expected[0] }} did not happen for {{ flow.name }} by {{ event.payload.deadline }}

All guards apply: once="per_run" keys on the arming run, and runs created by a lapse's run_flow action never re-trigger the same rule unless allow_self is set.

In the UI and the API¶

The rule form's Unless section has the same fields (unless, within, at, tz); validation rejects unless without either within or at. The rule page lists open expectations, GET /api/rules/{id}/expectations returns them (?open=false for history), and Test renders a synthetic lapse for a proactive rule.

Offline¶

Event-armed expectations are evaluated when the run ends: a run that overruns within fires the lapse at completion, provided the rule was registered by a server earlier. Clock-armed rules only work with a server.