Auto evaluation: test the output

Skills are reusable AI workflows that an agent runs with the organization’s own data. Auto evaluation tests each one, catches errors and suggests improvements, so skills stay accurate, current, and often better than builders expected.

Auto evaluation: test the AI workflow, not a document

Skills are reusable AI workflows that an agent runs with the organization’s own data, with people approving the work at checkpoints. Users told us they were losing track of their skills and what each one did, and they needed a better way to test them. Auto evaluation tests each skill with the organization’s own data. It catches errors and suggests improvements, so skills stay accurate, current, and often better than builders expected.
Role
Product designer
platform
B2B AI SaaS platform
Duration
6 weeks
setting
Product, Engineering, Pilot users, C-suite

Product screens are under NDA

Have the password? Enter it to see the screens throughout this case study.

context & stakes

Test the AI workflow, not a document

Skills are reusable AI workflows that an agent runs with the organization’s own data, with people approving the work at checkpoints. As teams built more skills, two needs kept coming up in feedback: knowing what each skill does, and a reliable way to test it.

I was the product designer on auto evaluation for six weeks, working with Product, Engineering, pilot users and the C-suite. I started by asking builders and Admins how they found, trusted and tested skills, then aligned with Product on what the platform needed.
business & user impact

A skill library is only valuable if people can rely on it

We designed for the ambitious expert, a senior marketer who would rather find a flaw than have someone else find it, and the quality guardian, a team lead who signs off on work they didn’t make and needs to see it work before saying yes.
Trust drives reuse
People reuse skills they trust. Tested skills get run and shared instead of rebuilt.
Quality at scale
Admins can’t read every skill by hand. Evals give them evidence to approve, and a record for every version.
Less waste
Broken and duplicate skills waste runs and tokens. Accurate skills protect the time and token savings skills promise.
problem & bet

Problems surfaced with real work on the line

A skill is meant to be picked up by anyone on the team and run with their own documents. That makes it hard to know it works until someone relies on it.
When is a problem found?
With evals: caught here
Before an Admin reviews it or anyone runs it
✓
✕
Draft
Evaluate
Admin review
Publish
First real run
Every run after
Before: found here
By whoever runs it first, with real work on the line
Skills are generic
A skill has no campaign of its own. Testing it with one example only proves it works for that example.
Failures show up late
A missing output or a broken data connection was found by the first person to run the skill for real.
Admins approved by reading
Reviewers could read the instruction, but not see whether the skill produced its expected outputs.
the bet
If the AI tests each skill with typical requests and the organization’s own data, and suggests fixes, we’d get fewer skills that stay accurate and current, and get better over time.
constraints

What we had to design around

The builder, the AI and the Admin each have a part. At first, skills with critical findings could still be published. We pivoted: builders can still send them to an Admin, but an Admin can’t publish until critical and major findings are fixed, by the builder or the Admin. Moderate findings can still be published.
No documents added
Test requests stand in for the jobs people will bring.
Real data, not mock data
Runs use the organization’s data through the skill’s own MCP connections.
Live version keeps running
Fixing creates a new draft. People keep running the live version until the new one is approved.
People decide
The AI suggests fixes. Builders choose whether to apply them, and only Admins publish.
three key decisions

Evals that check how a skill works, not just whether it breaks

Each decision kept the builder in control and the result easy to act on.
decision 01

Test the skill, not a document

Evals run two kinds of tests. Simulated runs: the AI writes typical requests, runs the skill on each with the organization’s data and checks the result against every expected output. Skill checks: the AI reads the skill itself to see if the instruction covers every output, the data connections are available and it’s clear when to use it. Builders already list their expected outputs, so there’s nothing extra to write.
decision 02

Every failure names a field, and suggests the fix

Each finding is graded Critical, Major, Moderate or Fully correct. A failure points to the field that caused it and suggests wording in Chat, so the builder changes one line, not the whole skill. Passing skills get suggestions too, so a skill can do more than the builder first asked for.
decision 03

Evals across the whole lifecycle, not one gate

Evals run whenever a skill changes or reaches the end of its lifespan. After an edit, the skill shows as not evaluated until its tests run again, and each version runs the same test requests so results compare like for like. Every skill card shows its status, passed, needs attention, out of date or never tested, so people can tell which skill to rely on before opening it.
outcomes

Evals caught errors, then made skills better

We ran a pilot with 80 builders. Each one ran evaluations on skills they had already made, some built with limited AI support and some with full AI support. Results are self-reported by pilot builders.
75%+
of each builder’s skills built with limited AI support had issues they rated severe
9 in 10
evaluations passed for skills built with full AI support, with only minor areas to update
4 of 5
AI suggestions improved the skill, as rated by builders
1–2 edits
to fix most skills, and most fixes grew the skill beyond what the builder first intended
what I’d do next

Prove tested skills mean fewer, better skills

The next step is proving that tested skills reduce duplicates and stay current across the library.
Fewer duplicate skills
Active skills per job, as teams update one trusted skill instead of making another.
Skills kept current
Share of live skills evaluated within their lifespan.
Failures caught early
Issues found by evals compared with issues reported after publishing.