Skip to content
TraceroAI

Know which stage of your RAG pipeline failed.

TraceroAI records every answer your pipeline gives, checks retrieval, grounding and the answer itself, and names the stage that broke. Each verdict says whether an LLM judge or a cheap estimate made it.

$ pip install traceroai
Read the quickstartView on GitHub
Judge accuracy
94–96%
Licence
Open source, Apache-2.0
Runs on
Your servers
trace 5f7bf901gpt-4o-mini · 1.68 s

Question

is money within 10 days refunded?

Answer

  1. Retrieval

    The context covers the 5 to 7 day refund window.

  2. Grounding

    1 of 4 claims isn't supported: the context never says 10 days.

  3. Answer

    It addresses the question that was asked.

Replayed from real traces in the example app, judged by an LLM.

Each mark is one answer, checked at retrieval, grounding and the answer: red where it broke, amber where the checks couldn't decide, hatched where only an estimate has looked.

Anatomy of a diagnosis

One real trace from the example app, start to finish.

  1. A question comes in

    Your app calls the SDK with the user's question.

  2. Retrieval

    Your retriever returns three chunks of the refund policy. TraceroAI records each one.

  3. Prompt and generation

    The chunks go into the prompt and the model answers. Every step is timed.

  4. The judge reads the answer

    It splits the answer into claims and checks each against the chunks.

  5. Probable cause

    One claim isn't in the context, so grounding failed: an unsupported claim.

  6. Recovery

    The agent regenerated with a stricter prompt. On attempt 3 the judge found every claim supported.

Three stages, one probable cause

A RAG answer can go wrong in three places: the retriever fetched the wrong documents, the model said something the documents don't support, or it answered a different question. TraceroAI checks all three for every trace and names the first that failed, so a bad answer becomes a lookup rather than a guess.

  1. 1RetrievalProbable cause

    Did the retrieved context match the question?

    Retrieval miss

    The documents that answer the question weren't retrieved, so the model had nothing right to work from.

    For example: Asked about annual-plan refunds; only the monthly billing pages came back.

    Usual fix: Retrieve more, or fix the index.

  2. 2GroundingNot the cause

    Is every claim supported by that context?

  3. 3AnswerNot the cause

    Does the answer address the question?

Healthy answer
All three checks passed.
Correct refusal
The documents don't cover it, and the model said so.
Needs review
The checks couldn't decide; a person should look.
The execution timeline of a trace: query, retrieval in 223 milliseconds, prompt, generation in 1.45 seconds and evaluation, with the generation step open showing the grounding check failed and the answer check passed.
Figure 1. In the dashboard: every step of a trace, timed, with what went in, what came out, and the check that judged it. Hover to look closer.

Every verdict says who made it

Checks run in three tiers. The cheap ones run immediately; the LLM judge runs in the background when you give the server a provider key. The dashboard draws a judge's verdict solid and calls it Final, and draws an estimate hatched and calls it Preliminary, so you always know how much to trust what you're reading.

CheckWhen it runsShown as
Term overlapOn ingest, in the request. No network calls.Preliminary
Embedding similarityIn the background, when a provider key is set.Preliminary
LLM judgeIn the background, when a provider key is set. Overrules the estimates.Final

Without a judge configured, every diagnosis is Preliminary and the dashboard says so. When the judge disagrees with an estimate, the estimate stays visible, struck through.

When a stage fails, retry that stage

The optional RecoveryAgent wraps your own retrieve and generate functions. After a retrieval miss it retrieves more; after an unsupported claim it regenerates with a stricter prompt. Every attempt is traced and grouped into one run, so you can see what failed, what the agent changed, and how it ended. By default it only retries on the judge's verdicts, not on estimates.

A recovery run in the dashboard: the original attempt and a retry were unsupported claims, the agent regenerated with a stricter prompt each time, and the third attempt was healthy.
Figure 2. One run, attempt by attempt, with the agent's decision between them.
pip install "traceroai[recovery]"
from traceroai.recovery import RecoveryAgent

agent = RecoveryAgent(
    client,
    retrieve=retrieve,    # (question, top_k) -> chunks
    generate=generate,    # (question, context) -> answer
    max_attempts=3,
)
result = agent.run("Is money refunded within 10 days?")
result["succeeded"], result["attempts"], result["trace_ids"]

Show that a change helps before it ships

Run the same labelled questions through each version of your pipeline, a different top_k, prompt or model, and let the judge grade every answer. The dashboard shows each variant's score and the questions where they disagree, which is where the change actually made a difference.

An experiment's question grid: ten questions graded for top_k 2, 3 and 5. The variants disagree on one question, which top_k 3 got wrong.
Figure 3. Ten questions, three variants: they disagree on one, and it's listed first.
run_experiment
from traceroai.eval import Case, Variant, run_experiment

result = run_experiment(
    client=client,
    dataset=[Case("refunds", "How long does a refund take?", "5 to 7 business days")],
    retrieve=retrieve,
    generate=generate,
    variants=[Variant("k2", "top_k=2", top_k=2), Variant("k5", "top_k=5", top_k=5)],
)
print(result.recommendation)

How often the diagnosis is right

We measured each tier against 117 labelled traces. The numbers below are the whole result, not a selection; the full report lists every verdict and how to reproduce it.

Diagnosis accuracy by evaluation tier
TierRight diagnosis
Term overlap
58.1% (68 of 117)
Embedding similarity
76.1% (89 of 117)
LLM judge
94.0–95.7% over 3 runs

The cases come from one small support-style knowledge base, and most of their labels follow from how they were constructed. Treat the figures as a comparison between tiers, not as accuracy on your production traffic.

What runs where

Everything runs on your machine or your servers: one Docker Compose file starts the API, the queue, the worker and the dashboard. Trace data leaves only if you configure an LLM provider for the judge, and then only to that provider.

  1. Your RAG app sends traces with the SDK to the API.
  2. The API runs quick checks, stores the trace in Postgres and queues it on a Redis stream.
  3. A worker runs embeddings and the LLM judge, calling your LLM provider only if you set a key, and writes verdicts back.
  4. The dashboard reads everything from the API.

Run it yourself

Start the stack, point the SDK at it, and send your first trace.

1. Start TraceroAI

Terminal
git clone https://github.com/chinmai-sd-123/TraceroAI.git
cd TraceroAI
docker compose -f infra/docker-compose.yml up --build
# API on http://localhost:8000, dashboard on http://localhost:3000

Add TRACEROAI_OPENAI_API_KEY to the environment to turn on the LLM judge.

2. Send a trace from your pipeline

from traceroai import TraceroClient

client = TraceroClient(base_url="http://localhost:8000")

with client.trace(question) as t:
    chunks = retrieve(question, top_k=3)
    t.log_retrieval(chunks, strategy="vector")
    prompt = build_prompt(question, chunks)
    t.log_prompt(prompt, version="grounded_v1")
    answer = llm(prompt)
    t.log_generation(answer, model="gpt-4o-mini")
# Each step is timed, and the trace is sent in the background.

Read the documentation for the full SDK reference, recovery options and API keys.