Skip to content

Documentation

Troubleshooting

Common issues and how to debug them.

Troubleshooting

Start from the Job page when you can. It shows the Workflow, each Step, each Run, the live transcript, and the diff Syrus captured.

A Run stays queued with host admission deferred

The queued-Run banner reports the admission reason, deferral count and start time. For host_resource_semaphore_busy, it also shows whether compute units, memory reservations, or measured memory reached the host budget. An idle worker always admits one Run unless its health sample reports critical memory, disk, or CPU pressure.

An immutable-source distributed grader rejected for critical local pressure or a full host resource guard can move immediately to another live, capability-compatible worker with healthy headroom. The banner diagnostics identify the rejecting worker, alternatives considered, and the selected routing decision. Reroutes use a hostname-specific queue, so they remain worker-specific even when multiple workers share storage. If all eligible workers are pressured, Syrus schedules a bounded backoff rather than repeatedly bouncing the grader between workers.

Fresh installs leave soft admission control disabled, so Docker Compose needs no CPU, memory-reservation, or staggering tuning by default. The guidance below applies when an admin has enabled admission control for a shared instance.

In Docker Compose, SYRUS_WORKER_CPU_LIMIT determines host compute capacity and therefore the fallback memory reserved per Run: effective memory divided by compute capacity, multiplied by the Run's capacity units. A low Docker Desktop CPU allocation can make that reservation large. Set the variable to the CPU budget actually available to the worker, and increase Docker Desktop's memory setting when the measured working set is genuinely close to the limit.

A chat turn remains active after a worker dies

Syrus reconciles chat turns whose worker process disappears. A turn lost when Solid Queue prunes a dead worker is retried with bounded backoff; a turn whose job failed normally is marked failed instead, clearing the active-turn state. The retry re-runs the same logical turn without adding a duplicate user message and resumes provider state when it was persisted before the crash.

While a turn remains active, the chat page also reduces its fallback polling rate from every 3 seconds to every 10 seconds and eventually every 30 seconds. This limits browser and server load if worker-death detection itself is delayed.

The poller never picks up my issue

Syrus polls GitHub issues about every five minutes, so first wait one poll cycle unless you used Poll now or delegated the issue from Syrus.

Check these in order:

  1. The repository is registered in Syrus under the same owner/name as

the GitHub repo.

  1. Repository polling is enabled.
  2. The GitHub issue is open. Syrus ignores closed issues and pull

requests during issue ingestion.

  1. The issue has the repository's trigger label. The default is syrus,

but each repo can use a different label.

  1. The issue does not have syrus-skip.
  2. If the issue should attach to an Epic, it uses the explicit

<!-- syrus-epic: #123 --> marker comment. Visible Epic: #123 prose is ignored for routing.

  1. The user's GitHub token is present and can read the repository.
  2. The GitHub credential is not currently rate-limited. Syrus delays

autonomous repository polls until GitHub's reset time when quota is exhausted.

  1. If the issue was labeled during downtime or another ingestion gap and was

not edited afterward, wait for the repository's full reconcile window. The default is 24 hours and admins can tune it with repository_full_poll_interval_hours.

  1. A worker is running the default queue, where

PollAllRepositoriesJob and PollRepositoryJob run.

To force a quick check from the UI, open the repository in Syrus and use the GitHub issues panel to delegate the issue. That adds the configured trigger label through the same GitHub token Syrus will later use.

See Issue Authoring for the complete label, Epic, dependency, and pending-reference contract.

In Kubernetes, check the worker logs:

kubectl -n <namespace> logs deployment/syrus-worker -c syrus-worker --tail=300

In Docker Compose:

docker compose logs --tail=300 worker

The agent never handles my failing CI check

Syrus reacts to completed GitHub Check Runs on PRs that belong to open Syrus Jobs. It does not react to arbitrary PRs unless they are attached to a Syrus Job.

Check:

  1. The PR was opened by Syrus or is associated with an open Syrus Job.
  2. The failing CI provider reports GitHub Checks for the PR head SHA.
  3. The check run is completed with a failed conclusion such as failure,

timed_out, action_required, cancelled, or stale.

  1. The user's GitHub token can read check runs.
  2. The Job has not already hit the rolling CI failure cap: three

ci_failure Workflows per 24 hours.

If the token lacks Checks read access, PR-comment handling may still work while CI-failure handling logs a permission warning.

The agent ran but produced no diff

This usually means one of three things:

  • The prompt did not ask for a concrete repository change.
  • The agent decided no change was appropriate.
  • The agent could not install dependencies, run tests, or write files in

the workspace.

Open the Run transcript. Look for the last tool calls and the final answer. If the agent spent most of the run discovering how to install the project, add a .syrus.yml prepare section:

prepare:
  - npm ci
  - npm test -- --runInBand

If the Workflow fails at the prepare Step, the agent has not started yet. Open the prepare Step details to see the failed setup command, its working directory, exit status, and the final output tail from the installer or setup command.

For cron Jobs, no diff can be a successful result. Scheduled prompts should explicitly say what counts as "nothing to do" so the agent can close cleanly instead of inventing work.

syrus checkout failed in a post-checkout hook

Post-checkout hooks run after the CLI has already switched to the Syrus branch. When a .syrus.yml hooks.post_checkout command exits non-zero, the CLI prints the failed command and exit code, then stops.

Check the command output first. Most failures are local environment issues: missing dependencies, a database that is not running, a failed migration, or a package-manager lockfile problem. Rerun the printed command from the repository root after fixing the local issue.

To inspect the branch without running hooks, bypass them for one checkout:

syrus checkout --no-hooks JOB-123

The checkout is not rolled back when a hook fails. If you bypass hooks, run any required local setup manually before testing the branch.

PR creation failed

Common causes:

  • The GitHub token cannot push to the repository.
  • The default branch configured in Syrus does not exist.
  • Branch protection or organization policy blocks branch creation or

pushes from the token.

  • A branch with the same Syrus branch name already exists and is not in

the state Syrus expects.

  • The agent produced no diff, so there was no PR to open.

Check the pr_open Step transcript first. Syrus logs the Git command and GitHub API error there. Also verify that the token-bearing push URL is constructed only at push time; do not paste raw logs containing tokens into chat or tickets.

How do I read the transcript?

The Job page renders each Run's transcript inline. The full transcript viewer parses retained Claude and Codex session JSONL into assistant messages, tool calls, tool results, and system events where possible. If the provider session is missing or truncated, Syrus includes the Run's JobLog rows as fallback transcript events.

Admins also get a full transcript viewer:

/admin/runs/<run_id>/transcript
/admin/runs/<run_id>/transcript/download

The inline transcript is stored as JobLog rows. Provider JSONL transcripts may be retained for debugging while failed or cancelled runs are still useful to inspect. Succeeded transcripts may be pruned.

For deployment logs, use the platform:

# Docker Compose
docker compose logs -f worker
docker compose logs -f web

# Kubernetes
kubectl -n <namespace> logs -f deployment/syrus-worker -c syrus-worker
kubectl -n <namespace> logs -f deployment/syrus-web -c syrus-web

The transcript you usually want is in the Syrus UI, not raw container stdout.

How do I cancel a Job?

Open the Job and click Cancel & close. Syrus cancels any active Runs and closes the Job thread. Follow-up polling for PR comments and CI failures stops for that Job.

If you want to stop one active Run but keep the Job open for a retry, use the Run-level stop action on the Job page.

For PR-level opt-out, add the syrus-stop label to the PR. The next PR poll closes the Job with reason syrus_stop.

How do I retry a failed Workflow?

Use the failed Workflow's retry action when it is available. Syrus reopens the failed Workflow and failed Step, creates a new Run, and continues from the same workspace.

Use Run again when you want a fresh follow-up Workflow on the same Job and branch. Add replay context if the previous attempt failed for a specific reason.

If the workspace has already been cleaned up, retry-in-place is no longer available. Use Run again or start over.

MySQL connection issues

In production, Syrus uses separate Rails database roles for primary, cache, queue, and cable. In small deployments these can point at the same MySQL server, but the environment variables still need to be available to both web and worker processes.

Check:

  • DB_HOST resolves from inside the container or pod.
  • SYRUS_DATABASE_PASSWORD is present.
  • The three ACTIVE_RECORD_ENCRYPTION_* keys are present, or

RAILS_MASTER_KEY is present for a deploy that keeps those keys in Rails credentials. Processes that touch encrypted credentials need one stable source of encryption keys.

  • MySQL is accepting connections from the web and worker network.
  • Migrations have run for primary, cache, queue, and cable databases.

In Docker Compose, the most common mistake is starting the web container before MySQL is healthy. Restart the web and worker after the database is ready:

docker compose restart web worker

In Kubernetes, exec into the web pod and run a Rails database check:

kubectl -n <namespace> exec deploy/syrus-web -- bin/rails runner 'puts ActiveRecord::Base.connection.select_value("select 1")'

If db:prepare reports errors for cache, queue, or cable while the primary database migrated, fix the missing database role instead of assuming the app is fully prepared.

Fresh production installs use MySQL migrations from zero. The development/test db/schema.rb is a SQLite dump and is not the source of truth for production database initialization.

Scheduled tasks are not firing

Check:

  1. The task is not archived.
  2. The task state is scheduled, not paused, auto_paused, or

fired.

  1. The owning user's Pause scheduling setting is off.
  2. The cron expression fires at most once per hour, is interpreted in

UTC, and produces a future scheduled time.

  1. The previous scheduled PR is not still open when

pr_pileup_policy is skip.

  1. The default queue worker is running PollScheduledTasksJob.

If the task repeatedly failed, Syrus may auto-pause it after the configured failure threshold. Unpause the task after fixing the underlying prompt, credentials, or repository problem.

The agent burned through my Anthropic or OpenAI credits

First stop the active work: cancel the Job or stop the active Run.

Then reduce future exposure:

  • Lower Max turns under Agent Settings. The default is 200.
  • Keep scheduled tasks on skip pileup unless parallel scheduled PRs are

intentional.

  • Avoid broad prompts like "clean up the repo." Ask for a bounded file,

failing test, warning, or subsystem.

  • Add .syrus.yml prepare commands so the agent does not spend turns

rediscovering dependency setup.

  • Disable repository polling while you investigate noisy triggers.

Repo-level and account-level dollar budgets are roadmap work. Until they ship, max-turns, cancellation, prompt scope, and scheduled-task policy are the practical controls.

Tailscale isn't connecting

Tailscale runs in its own privileged container, started by Plugin Runtime, not the worker. Most connection problems come down to one of these:

  1. The plugin is disabled, or has no auth key. Enable it from

Admin → Plugins and set auth_key (TS_AUTHKEY) under its config. With no auth key, Syrus never asks for a container at all — check Admin → Plugin Services for an error state on the tailscale row and read its message.

  1. plugin-runtime itself is down or unreachable (missing runtime).

Tailscale's container can only be started by the runtime manager. On Docker Compose, confirm the plugin-runtime service is up:

   docker compose ps plugin-runtime
   docker compose logs --tail=200 plugin-runtime

Admin → Plugin Services shows unavailable for every service, including tailscale, when the manager can't be reached — that is a different problem from the Tailscale container being unhealthy (next item), so check the state field rather than assuming.

  1. The container is running but unhealthy. Admin → Plugin Services

shows unhealthy past the health-check grace period. Use that page's Logs panel on the tailscale row, or:

   docker compose logs -f plugin-runtime   # the manager's view of the container

Then Restart it from the same page. A stale or revoked auth key is the most common cause — rotate it in auth_key and restart.

  1. The daemon is up but not connected. /admin/tailscale shows

daemon_running: true, connected: false. This usually means tailscale up is still authenticating, or the auth key was revoked — check the container logs (previous item) for the actual tailscaled error.

  1. Kubernetes / external installs. The runtime manager never runs

under Kubernetes, so Tailscale is always deployed externally there — you run the privileged workload yourself and point Syrus at it with SYRUS_PLUGIN_SERVICE_TAILSCALE_URL. Admin → Plugin Services is read-only in this mode; there is nothing for Syrus to start, stop, or restart.

See the Features page for the full picture and GET /api/v1/app/admin/tailscale/status for the machine-readable status payload.

An admin action failed or still shows old data

Admin mutation errors are shown next to the control that failed. Maintenance tasks, user controls, operator-console settings, and Work Unit visibility also refresh their data after a failure, which is especially useful when another operator changed the same record first.

The Processes page refreshes automatically while processes are active and asks for confirmation before sending SIGKILL. GitHub App installation refreshes show that synchronization was queued, then poll briefly for the new sync result. If either action reports a conflict, use the refreshed row or status before retrying.

The data volume is filling up

Syrus warns in the UI when the worker data volume (SYRUS_DATA_ROOT) is nearly full. On single-host Docker installs, that volume shares its disk with Docker's own image store, so the space is often consumed by old, unused Docker images rather than by Syrus data — superseded Syrus backend images left behind by updates are the most common culprit. The desktop app cleans those up automatically once an updated backend is confirmed healthy, but images from other sources still accumulate.

Check Docker's image store first:

docker image prune -a

Caution: that removes all images not used by a container, not just Syrus ones.

If the image store isn't the problem, inspect retained workflow workspaces under $SYRUS_DATA_ROOT/workflows and clean up workspaces left by finished Workflows. If cleanup is not enough, resize the data volume before clone, prepare, or landing jobs start failing.