Our evaluation agent draws charts, but never writes the numbers

Architect is the agent inside Rhesis, our open-source platform for testing LLM apps. It plans test suites, runs them and explains the results. Until now it explained results in markdown, so a question like "which topics are weakest across my last runs?" came back as a table you had to read through.

In v0.20 it answers with charts. The interesting part isn't the chart library. It's the rule we built everything around:

The model decides what to chart. It never supplies the numbers.

LLMs are good at choosing the right question and the right chart type. They are bad at copying 40 numbers out of a tool result without changing one of them. So we split the two jobs.

The design: the model writes a query, not data

The agent gets a show_chart tool. Its arguments describe what to draw, not the data:

{
  "query": {
    "entity": "test_result",
    "group_by": ["topic"],
    "measures": ["pass_rate"],
    "order_by": ["pass_rate"],
    "limit": 10,
    "min_count": 5
  },
  "chart_type": "horizontal_bar",
  "title": "Weakest topics",
  "x": "topic",
  "y": ["pass_rate"],
  "target": 90,
  "focus": ["Refund policy"]
}

The tool then:

1. Checks the request before running anything. Does x match a group_by dimension? Is every y a measure in the query? Does every focus value exist? A bad call is rejected with a message that lists the valid values, so the model can fix it.
2. Runs the query through the same insights API behind the dashboard, using the user's own credentials. A chart can't show data the user couldn't already see.
3. Builds the chart spec from the rows that come back, capped at 200 rows, and attaches it to the reply.
4. Gives the model a short text summary: the labels and values, with long charts cut down to both ends plus the true minimum and maximum. That's enough for the model to write about the chart without needing every row.

The frontend renders the spec with @mui/x-charts: line, area, bar, horizontal bar, stacked bar, scatter, radar, a shaded table for two-way breakdowns, and a single headline number. Every chart has a toggle that shows its exact numbers as a table, plus export to CSV, PNG and SVG.

What broke when we pointed it at real questions

The first version passed its tests and still gave bad answers. Here are the bugs that only real answers showed us.

The model invented category names. At first the summary only gave each measure's min and max. To describe the chart, the model made up topic names and printed them in a table right next to a chart of the real ones. It also read "min 40%, max 90%" as a trend. Fix: the summary now includes the actual labels and values.

The reply retyped the chart as a table. The model that writes the final reply had been told "use tables for numbers", didn't know a chart was being shown, and could see every charted value. On turns with a chart, it now gets a chart rule instead of the table rule. Tables of chart numbers went from 4 in 5 answers to 0 in 10.

Charts that were technically right and still misleading.

• Two pass rates stacked into one bar give an axis that runs to 200%. Stacked bars and areas now only accept counts.
• Cost and latency on one axis turned cost into a flat line at the bottom. Measures of different kinds can't share an axis.
• A title promising "Top 10 topics" over 5 rows. That title is now rejected.

Making a small, cheap model reliable

Architect runs on a small model, which can make more mistakes in tool calls, and we didn't want users watching it fail and retry. So instead of rejecting every imperfect call, show_chart fixes the ones it safely can:

• a stacked chart of rates is drawn side by side instead
• mixed measure kinds become one chart per kind
• duplicate charts are skipped
• a reply that ran a data query but finished without a chart is sent back once to add one

Try it

Rhesis is open source: github.com/rhesis-ai/rhesis. Charts ship in v0.20. Some questions to start with:

• "Which topics are weakest across my last runs? Ignore topics with fewer than 5 results."
• "How has the pass rate changed week by week?"
• "Compare our two endpoints across all metrics."
• "Show p95 latency and cost by day in production."

The Reports and Charts docs list every chart type and breakdown.

If you're building agents that report on data, the main lesson is the one we started with: let the model pick the question, and let your code fetch the answer.