<!-- Source: https://console.uubright.com/docs/reliability/ -->
# Failure and recovery

What survives when something goes wrong, and what doesn't. uuBright Console is built so
that **dropped connections and control plane restarts don't lose task activity**.
It does **not** make a task's actions exactly-once, and a crashed task loses what it
hadn't reported yet.

## How delivery works

- **Activity from a task** (Claude's messages, tool calls, approval requests, status)
  is numbered by the task's runner and kept in its memory until the control plane
  confirms it has written it to the task's log. After a reconnect, the runner resends
  everything unconfirmed, and the control plane skips numbers it already has.
- **Your actions** (follow-ups, approvals, interrupts) are saved in the task's record
  first, then sent until the runner confirms receipt. The runner ignores repeats
  during one run.
- **Your messages** (the first prompt and every follow-up) are also saved in the
  task's own state before the runner confirms them, and stay there until the agent
  finishes the turn they started. A message the agent never got is delivered when the
  task starts again. One whose turn was cut short is continued automatically when a
  crash or restart did it, and waits for you when you stopped the task (see
  [below](#what-a-crash-can-lose)). The runner remembers the messages it has
  finished, so a repeat after a lost confirmation doesn't run twice.
- **Queued follow-ups** carry stable ids. The runner records a message as sent on
  disk before handing it to Claude or Codex, then reports `message_started` over
  the acknowledged event link. A withdrawal is an acknowledged command: the
  runner removes only an unsent inbox entry, persists that decision before its
  ack, and reports `message_withdrawn` or `message_withdraw_refused`. If the
  message command has never reached a runner, the control plane removes it from
  the outbox directly. A reconnect replays unacknowledged commands and events.
- **The app** replays the task's log from where it left off when you reconnect.

## What happens when…

| Situation | Effect on the task |
|---|---|
| Your browser, phone or SSH tunnel disconnects | None. The task keeps running; the app catches up when you reconnect. |
| The control plane restarts or is updated | Tasks keep running. Their runners reconnect and resend unconfirmed activity; your saved actions are delivered. Open approvals stay open. |
| The connection between a runner and the control plane drops | Same as a restart: activity and actions are resent when it reconnects. |
| The runner or Claude Code crashes or exits | The task becomes **Stopped**, within about 10 seconds if nothing reports it. Open approvals are cancelled. If it was in the middle of a turn, it [starts again and continues the turn](#turns-cut-off-by-a-crash-or-restart). |
| Another process on the host kills the agent or the task's runner, for example another task's `pkill`, or the kernel's OOM killer | The feed shows **Agent stopped**: *killed by an outside signal*, with a **Resume** button. It appears when Claude Code or Codex ends with SIGTERM or SIGKILL (exit 143 or 137) without the runner having asked it to stop, or when a host task's runner gets SIGTERM or SIGHUP before any SIGINT (uuBright Console's own stops start with SIGINT). A runner killed with SIGKILL can't report it; the task shows **Stopped**. Nothing you sent is lost; see [below](#what-a-crash-can-lose). A task killed in the middle of a turn also [starts again and continues the turn](#turns-cut-off-by-a-crash-or-restart) on its own. |
| You select **Stop response** or **End session** in the task menu | The current turn stops, or the agent process exits. Open approvals are cancelled. The conversation is kept, and the task starts again when you resume it or send it a message. Nothing is continued automatically. |
| The server reboots | Everything stops. After you restart the service, tasks that were in the middle of a turn start again and continue it; idle ones stay stopped until you message them. See [Operate your server](https://console.uubright.com/docs/operations/#after-a-reboot). |
| A turn goes quiet: no output or tool calls for 20 minutes, and none of the agent's commands running | The task shows **Stalled** in Needs you. See [Stalled tasks](#stalled-tasks). |
| The agent hits a rate limit or its plan's usage limit | A request limit waits and retries after its backoff; a plan usage limit asks how to continue unless a workspace policy already chose. See [Rate and usage limits](#rate-and-usage-limits). |

**Resume** restarts the task's runner, which continues Claude Code's saved
conversation in the same workspace. A message to a stopped task does the same, then
delivers the message.

## Turns cut off by a crash or restart

When a task stops in the middle of a turn and you didn't stop or interrupt it (the host
rebooted, the runner was killed, Claude Code died), the control plane starts it again.
At most 3 tasks start at once on a host (`CCR_CLOUD_MAX_STARTS`); the rest wait their
turn. When the runner connects, the task gets one automatic continue message, ahead of
anything you queued. It says the turn was cut off, quotes the message the turn was
working on, and lists the turn's last 8 tool calls with their results, with "check state
before repeating any of these". Commands that were running, background jobs included,
died with the runner, so it also tells the agent not to wait for their notifications.
The feed shows one line: *Continued automatically*.

The continue message takes the place of the cut-off message, so that message doesn't
wait for you and the messages after it run in order. A turn is continued automatically
at most twice. If it is cut off a third time, the task stays stopped and shows
**Stopped unexpectedly** in Needs you, with **Try again**, **Restart fresh from branch**
and **Dismiss**.

A turn you interrupted (**Stop response**) or a task you stopped (**End session**) is
never continued, and a message held after your own stop waits for you as before.

A task that stops again while starting, before its agent takes a message (a runner that can't
start at all, say), is started again at most twice. After that it shows **Could not start** in
Needs you, with **Try again**. A start that fails gives its place in the per-host cap to the next
task at once.

Only you start a stopped task: by resuming it, or by sending it a message from the app, the API or
a chat channel. Messages the control plane sends on its own (a lead to its worker, PR feedback
routed to its task, a room passing a message on) are refused by a stopped task, as before. PR
feedback then waits in Needs you. A Stop while a start is still waiting for its place cancels that
start.

## Stalled tasks

A running turn with no output and no tool calls for 20 minutes
(`CCR_CLOUD_STALL_MINUTES`), while none of the agent's commands is running, is stuck.
This happens when the agent waits for a background job's notification and the job died
with a restart: the agent sits idle and an interrupt doesn't free it. The runner
checks the agent's child processes, not counting its MCP servers, so a long test run or
build in progress is never called a stall. The task shows **Stalled** in Needs you
(and as a decision on your phone) with:

- **Nudge**: interrupts the turn, drops what the runner knew of it (background tasks
  included) and sends a message that lists the last tool calls and asks the agent to
  check state and carry on. Try this first.
- **Restart fresh from branch**: commits any uncommitted work as a WIP commit on the
  branch the worktree is on (hooks skipped; nothing is reset, so the files stay as they
  are), ends the agent session and starts a new one in the same worktree. Its first
  message is a handoff note: the task's goal, the latest message, the branch, the last
  steps and the WIP commit. The old conversation isn't resumed.
- **Dismiss**: leaves the task as it is.

The card clears on its own if the agent produces output again.

## Rate and usage limits

When Claude Code or Codex can't get past a limit, the turn ends and the runner reports which kind:

- **Usage limit**: the plan's usage window is used up (Claude Code's *You've hit your limit · resets 3pm*,
  Codex's usage limit). The agent says when the window resets.
- **Rate limited**: the provider refused requests (HTTP 429) or was overloaded (529), and the agent's own
  retries didn't get through. Retries that do get through show only **Provider busy, retrying** in the feed.

For a plan usage limit, a decision card appears in the task and Needs you. It offers
continuation with the other agent, Claude API access when available, or waiting for the reset.
The owner can remember a choice per workspace. See [When an agent hits its usage limit](https://console.uubright.com/docs/tasks/#when-an-agent-hits-its-usage-limit).

When waiting is chosen, the feed says so in one line, and the task's header, its row in the task list and its board card count down:
**Rate limited · resets at 14:05 (in 23 min)**, or **Rate limited · tries again at 14:07 (in 2 min)** for a
request limit. The waiting task counts as Working. It then continues by itself, with a
message that says why the turn stopped and asks the agent to carry on. Its conversation is intact. Nothing is sent before the limit lifts, so no attempts are
spent on a limit still in force:

- After a usage limit, 30 seconds after the reset. Without a reset time, after 30 minutes.
- After a rate limit, after the provider's Retry-After if it gave one, otherwise after 1, 2, 5, then 10 minutes.

A usage limit belongs to the login, not the task: every task using the same login (this server's Claude
or Codex login, a workspace's own login, or the CCR gateway) waits for the same reset, and they continue
20 seconds apart, with a little randomness, so they don't all reach the provider at once. While a login is
limited, the New task form says so for the agent and model access you pick, with the reset time, before you
start the task (`GET /api/rate-limits` lists limited logins). A reply that gets through ends the limit for the task, and for a usage limit, brings the other
waiting tasks on that login forward.

Automatic continues are capped, so a limit that keeps coming back can't keep a task going on its own:

- A task you started is continued **once** per limit. A worker (started by a Manager or lead) is continued
  up to 4 times per limit.
- At most 6 automatic continues per task since your last message to it or its last turn that finished cleanly,
  however many limits they were for. A limit that lets one step through per try would otherwise never end.
- At most 20 automatic continues an hour across all tasks on one login.

At a cap the task shows **Rate limited** in Needs you, with **Try again** (continue now, and start the count
again) and **Dismiss**. A continue also waits while the task has a question, decision or form open for you. Needs
you shows a request limit only then; a plan usage limit asks immediately unless a workspace policy chose what to do. While a continue is scheduled, the task needs nothing from you. A message you
send meanwhile runs at once; if it gets through, the limit is over. **End session** cancels the scheduled
continue. Workers that are waiting on a limit are not nudged by the stall check.

## Long commands: durable jobs

A command run in the background (`&`, or Claude Code's background Bash) belongs to the agent's process,
so a restart kills it, and the agent can wait forever for a notification that never comes. For anything
that takes more than a couple of minutes, the agent uses **jobs** instead: the `ccr-task` tools
`job_start(cmd, timeout_s)`, `job_status(id)`, `job_wait(id, max_s)`, `job_log(id, tail)` and
`job_stop(id)`. Every task is told to, in its instructions.

- **Approval.** `job_start` runs a shell command and `job_stop` ends commands, so they are approved like
  one: in Ask mode they show an approval card, with the exact command for `job_start` ("Command, as a
  durable job"), and Codex asks the same way. Plan mode and dontAsk refuse them without asking. In Auto
  and Bypass they go ahead, like any command there. The other job tools only read and never ask. The
  task's socket, which the agent's shell can also reach, starts or stops a job only when you approved
  that exact call (one approval, one call), or in Auto and Bypass.
- **Survives restarts.** The runner starts each job in a process group and session of its own, so a
  crash of the runner or the agent, or an automatic restart, leaves the job running. Your **End
  session** (Stop) ends the task's jobs too, as does deleting the task. In Docker mode a container
  restart ends them.
- **Files.** Each job has a folder under the task's state directory (`jobs/<id>/`) holding the command,
  its output (`log`) and, when it ends, an exit file with the exit code. A log past 50 MB moves to
  `log.1` (the one before is dropped), so a job keeps at most about 100 MB of output. Only the 20 newest
  finished jobs keep their folders. The timeout (default 1 hour, at most 24) stops the whole group.
- **Its end, once.** When a job ends, the runner logs it in the feed and sends the agent one message,
  *[Job &lt;id&gt; finished]*, with the exit code and the last lines of the log. An idle agent starts a
  turn for it; a busy one gets it after the current turn. A job that ended while the runner was down is
  announced when it starts again, and only once. If the agent already read the end through `job_wait`
  or `job_status`, the message isn't sent, even when it was already waiting in the queue. A turn that
  message started and a crash cut off is [continued](#turns-cut-off-by-a-crash-or-restart) with its text.
- `job_wait` returns before the agent's MCP tool timeout (about 55 seconds by default), so the agent
  calls it again or ends its turn.
- The task's header shows running jobs beside its state ("· 1 job running"), with each command, how
  long it has run and a **Stop** button that ends the job and its process group. The agent is told it
  was stopped.

## Side effects that happen once

The tools that act outside the task take an optional `idempotency_key`: `room_post` and `task_add` in
rooms, `publish_artifact`, and a lead's `task_create`, `task_message` and `review_pr`. A second call
with the same key (within a week) returns the first call's result and doesn't post, publish or start
anything again; that is how an agent redoing its last steps after an automatic continue avoids acting
twice. Without a key a call acts even with the same arguments: telling a worker "Rebase on main" a second
time is a new action. The one exception is `publish_artifact`: the same file, unchanged, published again
within 2 minutes returns the first result, while a changed file always becomes a new version. A call that failed isn't
remembered, so it can be tried again. Pushes and `gh` commands the agent runs itself aren't covered.
The automatic continue lists them so the agent can check before running them again.

## A worktree on another branch

A host task works in a Git worktree on its own branch, `ccr/<task id>`. If the agent
checked out another branch there (for example to push to a PR branch), starting the
task puts the worktree back when that loses nothing: the tree has no uncommitted changes
and the task branch's commits are all in the current commit, or the task branch has no
commits of its own. The task branch then moves to the current commit and the worktree
switches back to it. The feed says what was moved, and that the agent should push with
`git push origin HEAD:<branch>` instead.

Otherwise (uncommitted changes, or the task branch has commits the other branch
lacks), nothing is moved. The task shows **Worktree on another branch** in Needs you
with the reason and the exact commands to fix it, and **Try again** once you have.

Each host task's runner leads its own process group and session; its agent and
the agent's commands run in that group. Stopping a task signals that group only
(SIGINT first, then harder signals if it doesn't exit), never by process name,
so stopping one task can't reach another. The group is recorded in
`process-group.json` in the task's folder: `kill -INT -<pgid>` stops that one task
by hand, the way uuBright Console does (a SIGTERM first is reported as an outside kill). Host tasks still share one Linux user, so agents are told not to use
`pkill`, `killall` or `kill -1`, and Claude Code and Codex are blocked from the
common forms
([security](https://console.uubright.com/docs/security/#what-uubright-console-enforces)).

## What a crash can lose

When a runner or Claude Code process dies unexpectedly:

- **Unreported activity.** Anything the runner hadn't yet had confirmed by the
  control plane is lost with it. On a normal exit, the runner waits up to 10 seconds
  to deliver everything first.
- **Nothing you sent.** Messages sent before the agent could start are delivered in
  order after you resume. The runner logs *Delivering a message saved before the task
  stopped* for each one.
- **Certainty about the interrupted message.** Claude Code and Codex don't save a
  prompt to their own conversation the moment a turn starts, so after a crash nobody
  knows whether the agent saw the message whose turn was cut short, or acted on it.
  When a crash or restart cut it short, the automatic continue message quotes it and
  lists what the agent did, so the agent checks before redoing anything. When you
  stopped the task yourself, after you resume it shows **This message may not have
  reached the agent** with **Send again** and **Dismiss**. Messages sent after it wait
  until you choose, so they stay in order. The task reads **Message needs a decision**
  in the sidebar, has a card in **Needs you**, and sends a notification, so it never
  sits idle unnoticed. Dismiss it if the agent already did the
  work; check the workspace first. The same happens when Codex doesn't answer a
  request to start a turn. If Codex refuses one outright, the message is sent again on
  resume.
- **Open approvals.** They're cancelled and shown as **Cancelled**. Claude never got a
  decision, so it may ask again after you resume.
- **The tool call in progress, and background jobs.** They aren't retried. Their side
  effects, such as edited files or a command that already ran, stay as they were. The
  automatic continue lists the last calls so the agent can check them, but it can't
  undo or deduplicate a push or a deploy that already ran.

## What isn't guaranteed

- **Exactly-once actions.** A decision can, rarely, be delivered twice: if a runner
  restarts after receiving it but before its confirmation reaches the control plane,
  the next run receives it again. A decision only applies to the
  exact request it answers, and a new run starts with no open requests, so a repeated
  decision can't approve something else; it fails harmlessly.
- **Exactly-once tool effects.** uuBright Console records what Claude did; it can't undo or
  deduplicate commands. That's why approvals show the exact command. Only its own tools that act
  outside the task [run once per key](#side-effects-that-happen-once).
- **Power-loss durability.** Logs are written as each event arrives but aren't forced
  to disk. A power failure or kernel crash can lose the last moments of activity.
- **Partial tool output.** Tool output is recorded when the tool finishes, not while
  it runs, so a long command shows nothing until it ends.

## Approvals

- A decision applies only to the one request it answers, in that task.
- A request can be answered once. A second answer, from another device or from
  Discord, is rejected.
- Interrupting, stopping, restarting or crashing a task cancels its open requests.
- **Allow once** runs the tool with exactly the input you reviewed.

## Next steps

- [Operate your server](https://console.uubright.com/docs/operations/)
- [How it works](https://console.uubright.com/docs/how-it-works/)
- [Troubleshooting](https://console.uubright.com/docs/troubleshooting/)
