flightdeck

A browser game writes a flight log. This pipeline reads that log, checks it against a written contract, and publishes the result as open Parquet.

1. Does the contract hold?

The pipeline drops nothing. A record that breaks the contract goes to quarantine with a reason. So the two parts must add up to the whole.

SELECT landed_records::INTEGER      AS landed,
       clean_records::INTEGER       AS clean,
       quarantined_records::INTEGER AS quarantined,
       reconciles
FROM contract_reconciliation

The pipeline computes that boolean and the page only reads it. The same check runs in CI on every commit, so a broken contract fails the build.

2. How fresh is this?

The page shows two times: when the site was built and when the data last arrived. srv is the arrival time, stamped by the server, so the page cannot write it.

SELECT records::INTEGER  AS records,
       sessions::INTEGER AS sessions,
       strftime(to_timestamp(first_srv / 1000.0)::TIMESTAMP, '%Y-%m-%d %H:%M') AS first_utc,
       strftime(to_timestamp(last_srv  / 1000.0)::TIMESTAMP, '%Y-%m-%d %H:%M') AS last_utc
FROM warehouse_manifest

3. Did anything get lost?

Every event carries an id, and the ids in one page load run in sequence. A gap there means an event never arrived.

SELECT events_received::INTEGER         AS received,
       ids_expected::INTEGER            AS expected,
       events_lost_by_id_gap::INTEGER   AS lost,
       peak_events_in_one_beat::INTEGER AS peak,
       cap_events_per_beat::INTEGER     AS cap,
       peak_pct_of_cap                  AS pct_of_cap
FROM loss_accounting

The contract caps how many events one beat may carry. The bar shows the busiest beat in the data against that cap.

4. Alarms raised

The recorder raises an alarm when it sees trouble in the page. The pipeline groups them by class.

SELECT class,
       n::INTEGER          AS alarms,
       page_loads::INTEGER AS page_loads,
       array_to_string(states_when_raised, ', ') AS states_when_raised
FROM alarms_by_class
ORDER BY alarms DESC

5. Errors caught

Every error the recorder caught, in arrival order, with the frame that threw it.

SELECT session_id,
       page_load_seq,
       strftime(arrived::TIMESTAMP, '%Y-%m-%d %H:%M:%S') AS arrived_utc,
       msg,
       top_frame,
       top_location
FROM error_report
ORDER BY arrived

Read Four clocks next. It shows how the pipeline tells a page that went quiet from a page that reported the wrong time.