Improving AI Skills with autoresearch & evals-skills
I’ve been trying to improve my AI skills using Auto Research, a library @karpathy shared for automatically improving AI prompts through repeated experimentation. I saw Ole on X share his fork of auto-research turned into a skill that is meant to tune other skills, so I decided to try that.
The idea is straightforward: define some test inputs, write judges that score outputs, let the optimization loop run, wake up to a better skill. I ran it three times before I understood what I was doing wrong.
Take One: Just pointed it at a skill
I picked new skills I’d built, handed them to Auto Research, and let the tooling do everything else. It generated the test inputs. It wrote the judges. It ran the optimization loop overnight. The scores went up almost immediately. It all looked great until I looked at what had changed.
The skills were far from improved. The problem wasn’t the tool. The issue was the criteria. They were machine-generated with no model of what real failure looked like. So the loop ran hundreds of experiments and got very good at satisfying those criteria. The skill got better at the wrong things.
Take Two: Wired in Hamel Husain’s eval skills
Hamel has done substantial work on the evals problem with @sh_reya. The skill for generating synthetic evals is more principled: you define dimensions of the input space (what feature the user wants, what persona they are, what scenario they’re in), then generate structured tuples across those combinations.
My inputs got genuinely better: more diverse, better coverage of edge cases. But I still left generation to the tooling. The inputs improved, but the LLM was still vibing it. I hadn’t personally read any outputs. I still hadn’t built any intuition from observation. The machine had better inputs to work against and still no real model of failure.
Take Three: Read the Evals Course Reader
I imported Hamel’s evals course reader PDF into NotebookLM and worked through it using the NotebookLM CLI in Cursor. From the course I remembered the Three Gulfs:
Gulf of Comprehension — gap between what you think your system does and what it actually does. What failure looks like, which cases break, in which ways, for which reasons. This gulf has to be closed before anything else can work.
Gulf of Specification — gap between what you want your system to do and what your judges measure. Direct consequence of skipping comprehension. If you haven’t seen real failure, you can’t write a judge that measures what matters.
Gulf of Generalization — gap between how the system performs on your test inputs and on unseen inputs. Auto Research’s optimization loop can address this — but only if the first two gulfs are already closed.
“If you are not willing to look at some data manually on a regular cadence you are wasting your time with evals.”
Error Analysis (Phase 1 of Analyze-Measure-Improve)
- Open coding — run skill on diverse inputs, read every output, write freeform notes on what’s wrong
- Axial coding — group notes into coherent failure taxonomy (binary, distinct categories)
- Write judges grounded in the taxonomy
- Validate judges — build mini golden dataset: manually score 15-20 outputs per criterion
- Then run Auto Research — and only then
The pattern: I kept wanting to skip comprehension and get to the automated part. It felt like moving faster. But I was just making the machine efficient at measuring the wrong things.
The Product Equivalent
PMs do the same thing with product decisions: skip the manual comprehension phase, jump to solutions or success metrics, measure hard against criteria that don’t reflect the actual problem.
The Gulf of Comprehension has a product equivalent: the gap between what you think users struggle with and what they struggle with. It closes when you’ve personally read enough customer conversations and interviews to develop a feel for what failure looks like.
“I don’t think you can automate your way past understanding. Someone has to close that first gulf, and in my experience, that someone is always you.”
원문: https://x.com/nurijanian/status/2035257434365976671?s=52