Agent Learning / Tool Tuner

We interviewed power users and reviewed session recordings of people attempting to improve tool outputs. The patterns were consistent:

Users could spot bad outputs instantly. The problem wasn't judgment. It was that there was no structured way to feed that judgment back into the system.

This shifted our framing. The design problem wasn't “help users write better prompts.” It was “let users teach tools through examples instead of instructions.” That reframe shaped every decision that followed.

We explored three different approaches before landing on the final design. Two didn't survive, but each one shaped the direction.

Rather than adding tool tuning as a settings page, we placed the entry point at the three moments where users are most likely to encounter a bad output and want to fix it.

The first step asks users to review past tool executions and rate them as good or bad. This is the core interaction: you don't need to understand prompting to know whether an output is right or wrong.

One deliberate friction: users have to rate an output before they can select it. This ensures every selected execution carries a quality signal for the tuner, not just a user's arbitrary pick.

Based on the collected ratings and feedback, the auto-tuner optimises the tool configuration across five areas. The user sees the result as a reviewable diff, not a black box.

We chose to show the full configuration diff rather than just applying changes silently. In testing, users who couldn't see what changed didn't trust the output. Showing the diff built confidence. Even when users didn't understand every individual change, seeing that changes were specific and scoped (not a total rewrite) made them comfortable proceeding.

The tuned configuration regenerates outputs from the selected past executions as temporary previews, not replacements. Users see the improved outputs alongside accuracy scores and can toggle between the original and improved version.

This step exists because of a failure in an earlier iteration. When we designed a version that auto-applied changes without a verification step, users reverted almost immediately. They didn't distrust the tuner. They distrusted not being able to check its work. Adding verification changed the save-through rate completely.

Saving applies the optimised configuration directly. The updated prompt, variables, and settings are written to the tool. All future executions use the tuned setup.

We kept this step intentionally minimal. By this point, users have reviewed the config diff and verified the improved outputs. The save action should feel like a confirmation, not another decision.

Some users didn't know how to start tuning. We assumed the entry points were self-explanatory, but early feedback showed that users weren't sure what “tuning” meant or when to do it. The flow needed a guided onboarding step — a short walkthrough that set expectations before asking users to rate outputs.

The UI felt too cluttered. We tried to surface everything at once — ratings, execution history, config details, and tuning controls — all in the same view. Users told us they didn't know where to look or what to do next. We should have structured it as a guided flow with clear steps instead of a dense dashboard.

Gating it as a pro feature limited our learning. Because auto-tuning was shipped behind a pro paywall, we had a smaller pool of users to learn from. Fewer people meant slower feedback loops and less confidence in our iteration decisions. In hindsight, we would have opened it up earlier — even in a limited form — to validate the experience with a wider audience before locking it down.