Four clocks

A page reports four clocks. The page writes three of them. The server writes one.

   wall, perf, timestamp  -->  written by the page   -->  claims
   srv                    -->  written by the server -->  fact

The pipeline orders every record on srv. It treats the other clocks as claims, and checks them.

1. Do the clocks agree?

srv minus wall, over every clean record, split by the origin that served the page.

SELECT origin_kind           AS origin,
       records::INTEGER      AS records,
       skew_min_ms::INTEGER  AS min_ms,
       skew_mean_ms          AS mean_ms,
       skew_max_ms::INTEGER  AS max_ms
FROM clock_skew
ORDER BY origin

A browser on the same machine as the recorder has almost nothing to disagree about. A browser reaching a server over the internet carries the trip in its skew. Reporting one number over both would describe neither, so the pipeline splits them and the page prints what it is given.

2. Gaps between arrivals

A beat arrives every second. The pipeline reports every gap longer than three seconds, measured on srv so the page cannot hide it.

SELECT session_id,
       page_load_seq::INTEGER      AS page_load,
       gap_s,
       prev_vis || ' -> ' || vis   AS visibility
FROM srv_gaps
ORDER BY gap_s

The pipeline explains each gap from vis alone.

SELECT * FROM gap_explained

A gap with the tab visible on both sides has no throttling explanation. The pipeline labels it and keeps it visible, rather than dropping it.

3. Time to detection

Every error and every alarm, in arrival order, with the seconds since the previous incident in the same session.

SELECT session_id,
       page_load_seq::INTEGER                                         AS page_load,
       strftime(occurred_at::TIMESTAMP, '%Y-%m-%d %H:%M:%S.%g')       AS occurred_at_utc,
       event,
       detail,
       s_since_previous
FROM incident_timeline
ORDER BY session_id, occurred_at

Read s_since_previous on an alarm row that follows an error row. That is the time the recorder took to notice a dead loop.