heka
Menu

Notes from the factory

Measure the change before you review it

An agent should make its work easy to take in. Plain words are half of that. The other half is a few numbers that tell a reviewer how big a change is and how hard to look.

Two docks on the water, side by side, and the same change lands on each as crates, one crate a file: a small group of 11 and a big group of 30. On the left dock, a reviewer opens every crate in order, the small group first, and each opened crate turns green. On the right dock, words go up before anything is opened: skim over the small group, read over the big one, and deep over two crates in it, which turn ochre. That reviewer walks straight to the small ochre crate in the middle of the big group and opens it, a finding, then the other deep crate, then reads the rest quickly. At the end the right dock is done, and the left reviewer is still opening crates and has not reached the small one.

Lately I have been building one idea into our agents: they are responsible for the cognitive load of whoever reads what they produce. That reader is often me, reviewing a pull request or a plan. We have not turned the idea into a skill yet. We have a couple of working drafts, and we are far from the only people working on it.

Plain words are the first half

A lot of people attack this problem through language. Matt Pocock’s wait-what is for the moment an agent’s message does not land: it tells the agent to pitch it again with a little context, in Simplified Technical English, using the project’s own glossary. Lauren Tan’s bro asks the agent to restate its last message “like one human talking to another”.

The language an agent uses decides how much work its output is to read. I think even caveman, which was built to save tokens by making the agent talk like a caveman, makes it easier for a person to see what is happening.

Numbers are the second half

The other part of reducing cognitive load is metrics: a few numbers that let a person take in the impact of a change quickly, especially the impact its title does not show.

I already ask for this when I start an implementation. Before any code is written, I ask an agent for a gap analysis, a chart, a breakdown or a rubric. A table of what exists, what is missing and what the work will touch is faster to read than three paragraphs about it. The same is true on the other side, when the work comes back for review.

Read the metrics before the code

Before we review a large change, we read its metrics first. They tell us the scale and the impact of what we are about to review. They also tell us how deep and how thorough the review should be: a skim, a careful read of every file, or a line-by-line review with a second reviewer.

When I say “we”, I don’t only mean people. An agent that reviews a change needs the same answers before it starts, and it can read the metrics the same way.

A skill collects those metrics for us on each large change: change-metrics, in our public skills repository. It is our version of Warp’s merge-risk, which builds on Matt Pocock’s pr skill.

First it measures the diff: files and lines, the areas it touches, the public interfaces it changes, the sensitive areas such as authentication, migrations, dependencies and build steps, and which checks ran. Then it rates the change on six categories: cohesion, alignment, impact, verification, rollback cost and error detection. Last, it picks a review depth and names the files to read first.

What it found in one of our own changes

We ran it on a recent pull request in the repository behind this site. It held 39 files across this site and heka news, in two commits. The first added two skills from other authors, one for page layout and one for readability. The second was a pass over the layouts of both apps and the wording of five posts, based on what those skills found.

The skill rated the change medium risk. It said to skim the manuals in the first commit but read the two outside skills closely, because they are instructions our agents will follow. It said to read the second commit, and to go deep on two files in it: the site’s markdown config, which gained a step of our own, and a small fix to the code that shrinks the CSS for heka news before it ships.

The CSS fix sat in the middle of 30 files of layout work. The old code deleted the space in .prose :not(pre) > code, so a style we had written never reached a reader. A build step decides what every page gets, and a two-line change to one hides easily inside a commit titled for something else.

Grouped pull requests keep the signal strong

How we use the skill depends on how we group work. We advocate for grouped pull requests. As we stack work into a pull request, we keep its change sets to similar areas, the same functionality and the same type of implementation. The skill reads those groups from the commits or the layers of the stack, and rates each group as well as the whole.

That is why the numbers still mean something on a large pull request. One risky group and two simple ones get one deep review and two skims, not three deep reviews because the total looked big. When a group mixes kinds of change, the skill marks it down under cohesion. On that same pull request, the second commit got “mostly” instead of “one kind”, because the CSS fix and the edits to the posts rode along with the layout work.

Try it

The skill is one file. Install it with npx skills add heka-vault/skills --skill change-metrics, or copy its folder. Then edit the list of sensitive areas at the top to match your own codebase.

If you want to learn more about how we do stacked pull requests, get in touch.

Tell us what should exist.

Bring us something you want built, or something painful you want gone. Talk to our agent now, or email us and an engineer answers.

A voice call with our AI agent.

Email us