Observability

Syrus records several kinds of operational evidence. The goal is to make debugging possible without shelling into pods first.

Performance

Admin Performance shows slow browser traces, requests, jobs, SQL statements, phases, and event writes for the current revision. Request details can group SQL under the request and under the deepest active phase, so a slow endpoint is traceable to the actual work it performed. Browser traces include both passive observations (long tasks, slow input, event-loop lag) and explicit frontend markers that frontend code places with the performanceMarkers API — named spans like diff_review.parse_diff or diff_review.syntax_highlight that break a slow UI surface down into the specific phase responsible, the same way backend phase events do for requests.

Use it when:

• a page takes seconds to load,
• a background job blocks throughput,
• a SQL statement appears repeatedly,
• a frontend interaction feels slow.

While at least one background job is active, each worker process also samples process RSS and container memory once per second. Crossing a new 256 MiB bucket or 70%, 80%, or 90% of the container limit writes a compact event directly to durable storage, including Ruby heap live slots and the complete set of active job contexts. Normal completion adds start, end, and peak memory readings for the job. The events deliberately show shared container memory alongside per-process RSS: concurrent jobs share a Ruby process, and separate queue-pool processes contribute independently to the same container total.

For SQL rows, the admin API can run EXPLAIN and safe EXPLAIN ANALYZE against captured statements when the query is safe to analyze. The UI presents the plan in a visual modal for easier index and join diagnosis.

Operational Logs

Operational logs are searchable, structured events for normal system activity: worker actions, sidecar events, plugin calls, preview lifecycle events, and other runtime notices. They are useful when nothing crashed but behavior still looks wrong.

Readiness, Metrics, and Alerts

/up is the lightweight Rails liveness check. /readyz is the unauthenticated readiness probe for deployments: it checks database boot, registered Solid Queue workers with fresh heartbeats, pause state, writable storage, and instance-level GitHub App setup without requiring an operator session. The Docker image and Docker Compose web service use /readyz for their health check.

/metrics exposes aggregate Prometheus metrics. Syrus ships canonical Prometheus alert rules in config/prometheus/syrus-alert-rules.yml for queue age, open provider circuits, stale recurring jobs, and a growing attention backlog.

Alarm-severity SystemAlerts can also leave the browser through SYRUS_ALERT_WEBHOOK_URL and/or SYRUS_ALERT_EMAIL_TO. A recurring background job evaluates the same alert pipeline every minute, so delivery does not depend on an operator having the app open. Delivery is deduplicated by dismissal key so one incident does not flood the outbound sink.

The optional Alertmanager Investigations plugin closes the loop in the other direction: it polls Alertmanager for active, unsilenced, uninhibited alerts and files a read-only Syrus investigation only when the alert's runbook annotation resolves on the repository's default branch, matches an operator-configured repository/path allowlist, and contains an explicit autonomous-investigation front-matter marker. Alert payloads are treated as bounded, untrusted evidence. Alerts without an eligible runbook are ignored, and the plugin deduplicates by alert fingerprint while preventing two open investigations for the same host. If the runbook reaches a decision or remediation point, the investigation report escalates with a reason code instead of performing the action.

Browser Errors

Browser errors capture client-side exceptions with path, user agent, revision, stack, and request context. They are separate from performance data because a fast page can still crash in the renderer.

The admin log UI can file a Syrus Job directly from a browser error and attach the full log entry to the prompt, so the agent starts with the evidence that was captured.

Backend Exceptions

Backend exceptions capture request and job failures with class, message, backtrace, runtime context, and revision. Treat them as the first place to look when the UI shows a 500.

Activity and Reconciler Logs

The activity log records important state transitions and why Syrus made them. The reconciler log records repair decisions for stale runs, orphaned queued work, worker-death recovery, closed-job cleanup, and similar consistency work.

Use these logs when a Job seems to move by itself, when a workflow is retried, or when a queued item disappears.

Live MySQL

The optional MySQL admin plugin is for installations that use MySQL instead of SQLite. When enabled, it shows live process list information, connection state, slow log status, statement digests when permissions allow it, and guarded query kill actions.

This view runs live MySQL commands on demand. It is not a replacement for retained performance logs.

Retention

Operational evidence is retained for bounded recent debugging, not forever. Exact retention can vary by deployment and log stream. The admin UI should be treated as a live incident tool; copy durable conclusions into Jobs, PRs, or docs when they matter long term.

Admin Retention Settings lets an operator configure how long each prunable table's rows are kept (per table, in days or hours; zero means unlimited) and see current and estimated size against available disk. For tables where historical data has forensic value (diagnostics, audit trails, agent session transcripts), an "archive before delete" toggle serializes each pruned batch to a compressed file before it's removed, instead of discarding it outright. Past sweeps for an archiving-enabled table can be browsed and downloaded from the same page — this is a download-only history: there is no built-in way to restore an archived batch back into the live table.

Practical Debug Paths

For a slow page:

1. Open Admin Performance and find the request path.
2. Inspect grouped SQL and phase timing for that request.
3. Explain the slowest SQL if the statement is safe.
4. Check Browser traces if backend time is low but the page still feels slow.
5. For a specific slow UI surface (e.g. the diff review tab), look at the

grouped browser traces named for that surface (diff_review.*) instead of only the generic browser.slow_input entry -- they share the same path, app revision, and timestamp window, so a slow input event on /jobs/4348?tab=review can be lined up against nearby diff_review.parse_diff/diff_review.syntax_highlight/ diff_review.viewport_render markers to see which phase actually dominated.

For a stuck Job:

1. Open the Job workflow tab.
2. Read the queued-Run banner. It distinguishes a saturated worker pool from

an unconsumed queue, a concurrency lock, a future schedule, a claimed job, a missing RunJob, no live workers, and a workflow start block. Use its link to inspect Admin → Queue → Pending or Workers; an unknown diagnosis is intentionally neutral and does not imply that the pool is busy.

1. Check the latest workflow, Work Unit status, and pause reason.
2. Read the activity and reconciler logs around the same time.
3. Inspect provider availability and admission-control status if the work is

waiting rather than failed.