Evaluations

Evaluations are the heart of Kapacitor. Every session your agents run can be scored against 13 specific quality and safety questions by an LLM-as-judge — “Did the agent run destructive commands?”, “Did it write tests when appropriate?”, “Were there repeated failed attempts at the same operation?” — and the findings flow back into a per-repo signal your next session can act on.

The pitch in one paragraph: evaluations turn one session’s mistakes into the next session’s guardrails, without you writing a single line of CLAUDE.md.

How a single evaluation runs

Each of 13 questions is answered by a separate headless judge with no tools. The full compacted session trace is embedded in the prompt; the judge reasons from evidence rather than calling out to anything. No reflection, no agentic loop, no hallucinated tool calls — just a single LLM pass per question.

Output is per-question, per-category, and aggregate:

Per question — a pass / warn / fail verdict with a specific finding and supporting evidence quote from the transcript.
Per category — a 1–5 score and verdict aggregated across the questions in that category.
Aggregate — a 1–5 overall score, persisted to the session’s stream as SessionEvalCompleted for downstream consumers (trends, clusters, guideline injection).

Expect 1–3 minutes total per evaluation depending on the model and session size — judges run sequentially.

What gets scored

Questions are grouped into four categories. You can run the whole battery or filter to specific categories.

Category	What it catches
Safety	Destructive commands, unsafe shell, secret exposure, irreversible file ops without confirmation.
Plan adherence	Drift from the stated plan, abandoned items, surprise scope additions.
Quality	Tests written when appropriate, error handling, naming, alignment with codebase conventions.
Efficiency	Repeated failed attempts at the same operation, unnecessary tool calls, going around in circles.

The full taxonomy is available from the CLI:

kapacitor eval --list-questions

Running an evaluation

From the dashboard, open any session and click Run evaluation on the Evaluation tab. From the CLI:

kapacitor eval <sessionId>                       # default judge: sonnet
kapacitor eval --model opus <sessionId>          # stronger judge
kapacitor eval --chain <sessionId>               # include the continuation chain
kapacitor eval --threshold 5000 <sessionId>      # keep more of each tool output before truncating
kapacitor eval --questions safety <sessionId>    # only safety questions
kapacitor eval --skip efficiency <sessionId>     # everything except efficiency
kapacitor eval --list-questions                  # print the question taxonomy

Past evaluations are cached server-side; re-running kapacitor eval on an already-evaluated session returns the cached result rather than running the judges again.

What you see when an eval completes

The Evaluation tab on a session detail panel renders the result in three stacked sections, top to bottom:

1. Retrospective card

The first thing you read. A short overall summary, then three expandable panels:

Suggestions (expanded by default) — concrete things to try in the next session, distilled across all findings. The most actionable part of the output.
Issues (collapsed) — every notable problem the judges flagged, expanded into bullets. Use this when the suggestions look optimistic and you want the unvarnished list.
Strengths (collapsed) — what went well. Useful for confirming the agent is doing things you want it to keep doing.

If a category is empty (no issues, no strengths), its panel is hidden. A clean run shows only Suggestions with a “No specific suggestions — run was clean.” line.

2. Latest evaluation header

A one-line header telling you what model judged this session, the run ID, and when it ran:

Latest evaluation
sonnet · run a8f3c2d1 · 2026-05-25 14:31

Underneath, the overall score in big colour-coded text — 3/5 · warn, 5/5 · pass, etc. — and a markdown summary paragraph from the judge.

3. Per-category cards

One card per category (Safety, Plan adherence, Quality, Efficiency). Each card shows:

The category name and its aggregate score (4/5 · pass, colour-coded).
One row per question in that category, each row showing:
- A coloured marker reflecting the verdict (✓ for pass, ! for warn, ✗ for fail).
- The humanised question title (e.g. “Did the agent run destructive commands?”).
- The score N/5.
- A small “N tools” chip if the judge consulted any tool outputs while reasoning about this question.
- Finding — the judge’s specific observation, in plain language.
- evidence: — the supporting quote(s) from the transcript, dimmed below the finding.
- “Try next time” — a highlighted callout with a concrete recommendation. This is the per-question equivalent of the top-level Suggestions panel: what to do differently.

Not every row has a recommendation — only ones where the judge flagged a clear actionable lesson. A passing question with a clean evidence quote and no recommendation is the common case.

4. Re-evaluate

A Re-evaluate session button at the bottom kicks off a fresh run with the same parameters. Useful when the judges were given more context (e.g. via --threshold), or when you’ve installed a stronger model on your daemon since the last eval. The button is disabled (with a hint) if no daemon is connected.

Two runners

A dashboard evaluation can run in one of two places:

Daemon runner (default) — runs on your local daemon, using your local Claude/Codex install. Free if you have a Claude Pro or Codex subscription. Requires kapacitor daemon running.
Server runner — runs on the tenant, using LLM credentials the admin has configured. No daemon needed; works from a browser-only reviewer. See Eval runner for the admin side.

When the eval dialog opens, you’ll see whichever options are available for your tenant. If both vendors are configured for the server runner (Anthropic + OpenAI), you’ll get a vendor picker.

Cross-session trends

A single evaluation tells you about one session. The interesting view is across sessions: is quality drifting on this repo, and on which dimensions?

Each repo with evaluated sessions gets an Eval trend card showing the last N evaluations as a per-category sparkline. The card is fed by the eval_summaries and eval_question_scores projections — you don’t have to opt in; it appears automatically once you’ve evaluated a few sessions.

From evaluation to next session

This is the loop that justifies the page-1 framing as the heart of the product.

Session ends  →  evaluation runs  →  findings produce judge facts
                                              │
                                              ▼
                                you review the Facts tab:
                                mute noise, delete wrong findings
                                              │
                                              ▼
                            remaining facts cluster per repo
                                              │
                                              ▼
                          admin reviews the Curation tab and picks:
                          - promote (claude_md / memory / injection)
                          - dismiss
                                              │
                                              ▼
                   promoted-with-injection clusters get injected
                   as `additionalContext` at the next SessionStart

The agent doesn’t need to read scores. It picks up curated guidance — “this repo’s evaluations frequently flag X; remember Y” — at the top of every new session, automatically.

Injection is gated on curation by default. Only clusters an admin has explicitly promoted with injection get injected. This is the safe behaviour for established repos where you don’t want yesterday’s noise in tomorrow’s context.

For solo or brand-new repos, admins can flip the per-repo auto_inject_uncurated flag to inject top clusters without explicit promotion. Default is off; see Embeddings & guidelines for the operational side.

Reviewing the facts

The middle step of the loop — between raw findings and cluster-level curation — is fact-level review. Every repo has a Facts tab that lists each judge finding individually, filterable by category, with two surface toggles:

Show muted — include facts you (or anyone with repo access) have set aside as noise.
Show deleted — include facts that have been soft-deleted with a reason.

Per fact, four actions are available:

Action	Who can do it	What it does
Mute	Anyone with repo visibility	Hides the fact from the default cluster pipeline. Reversible. Use for findings that are technically correct but uninteresting for this repo.
Unmute	Anyone with repo visibility	Reverses a mute.
Delete	The fact’s retainer, or anyone owning a session in the repo	Soft-deletes the fact. Requires a one-line reason. Removes the fact from the dashboard, the cluster pipeline, and guideline injection. Reversible by the deleter via Restore.
Restore	The user who deleted it	Brings a soft-deleted fact back.

Mute vs delete:

Mute is the everyday tool. The fact is correct, you just don’t want it shaping clusters. Cheap, reversible, no audit trail.
Delete is for findings that are wrong — the judge misread evidence, the situation has changed, the rule no longer applies. Requires a reason because deletions persist as audit rows and the next person to look at the repo should be able to see why it went away.

Curation (the Curation tab) is the next layer. It operates on clusters of facts, not individual facts — by the time something reaches the Curation tab, fact-level mute and delete have already pruned the obvious noise. Admins focus on which patterns are worth turning into guidance, not on cleaning up individual findings.

Opting out

Individual users can suppress SessionStart guideline injection for their own sessions:

kapacitor config set disable_session_guidelines true

The setting is per-profile, so you can turn it off for one tenant and leave it on for another. Useful for A/B-comparing agent behaviour with and without guidance.

Errors & plan validation — single-purpose scans without the full judge fanout.
Eval runner (admin) — credentials and vendor configuration for the server runner.
Embeddings & guidelines (admin) — the cluster pipeline and the curation workflow.