I like customizing my agents. Skills, plugins, extensions, whatever the particular tool calls them. But I want to combat the slow creep towards the dumb zone, and loading a bunch of unrelated stuff into every session works against that.

Matt Pocock has a useful description of the smart zone and the dumb zone. Early in a session, the agent is at its best. The longer the conversation goes, the more mistakes start to pop up. This is where those early-days patterns of hallucinations really came from. The context window isn’t a promise that it’ll work well at all times.

There’s no universal line where this happens. Maybe 40% to 60%? It depends on the model. But every model shares the fact that the more things you pack into its context, the worse it gets over time.

What are we adding?

Context is the available space the model has when it produces a response. Your conversation is part of it, but so are the base instructions from OpenAI or Anthropic, and information about the tools it can use. This is typically referred to as the system prompt. It supplies instructions before you’ve even typed anything.

Installing something extra doesn’t necessarily put all of it into the system prompt. For example, Claude Code only loads a skill’s full instructions when it’s used, rather than loading every skill in full at startup. This is great, but dozens of skill descriptions and available tool definitions can still take up context, depending on how the application loads them.

Keep the occasional stuff local

My recommendation is to start with nothing and only add things when you have a reason to use them. Don’t just add the Google Calendar support because you might use it one day. Now it’s in your system prompt for every project, creeping you closer towards the dumb zone.

However, if a tool helps with your everyday work across projects, keeping it in your global setup makes sense. If it’s for deploying one particular website, keep it with that website. A specialized database integration doesn’t need to be available while you’re editing an unrelated Python package.

For a concrete example, Claude Code supports personal skills in ~/.claude/skills/ and project skills in .claude/skills/. The first makes a skill available across your projects. The second keeps it in that repository. Use the equivalent project-level setup in your agent, and check the scope when you install something.

Personally after writing this I cut about 10 different writing and planning skills from my globally installed skills and moved them all into a writing project. Try it out yourself!

It had been done in the previous version in MATLAB, but I was less familiar with the inner working of PyLossless at the time. I was also a complete novice when it came to things like Qt 5, PyQt, and the mne-qt-browser.

To be clear, trying to build this system would require knowing the PyLossless pipeline state, MNE’s ICA tools, the MNE Qt browser, Matplotlib figures, file watching, and the actual EEG-cleaning logic. I had already failed a few attempts at this before because I just got tired of there being too many things to juggle.

On the suggestion of a friend, I made a new repository called pilot-qc and decided to throw some AI at it and see what would happen. Worst case would only be another failure.

Let the prototype be ugly

There wasn’t much of a plan beyond getting something on the screen.

The commit history tells the story pretty well. The first two commits landed on March 21. One was called first commit. The next was wow this is amazing, which is honestly a pretty good record of how I felt at the time. Later that day, I had figures communicating with each other, scalp snapshots for a selected time window, a history of component rejection, and raw and cleaned signals appearing together.

The prototype used a file named .local_reject to pass state between the plots. It stored a selected time range and a set of rejected ICA components. A Qt timer checked the file every half second. When something changed, the code copied the original recording, applied the current ICA exclusions, cropped it to the selected window, and plotted the cleaned channels over the raw channels.

Would I design a finished application this way? Absolutely not. But it answered the question I actually had. Could a reviewer change the rejected components and see what those decisions did to the EEG? All in Python and with the mne-qt-browser? Yep.

AI made the scratch repository more useful

This was early in my adoption of AI coding tools. The repository description was pretty clear about it: “Trying some AI assisted QC figure generation.”

I knew what I wanted from the EEG review process. The plotting and GUI code was another story. AI made it cheap to try another event handler, connect two figures, or rebuild a plot when the first attempt did something weird. Instead of staring at documentation and trying to design the whole thing in my head, I could run some code and complain about what it did.

The scientific judgement still came from me. An agent could connect a click to a plot update. It couldn’t decide which comparison would actually help someone judge whether removing an ICA component improved the recording. AI got me through unfamiliar implementation details quickly, and the throwaway repository kept the resulting mess away from PyLossless.

This worked much better than asking for a polished feature all at once. Generated code was allowed to be awkward in there. Anything I did was just a throwaway anyway.

Moving the useful part into PyLossless

The prototype didn’t become a package or a long-lived dependency. I had briefly wondered in the PyLossless issue whether it should be a submodule. Thankfully, I talked myself out of that one.

I moved the useful parts into PyLossless instead.

On April 10, about three weeks after the first experiment, I opened and merged PyLossless PR #29. It added the initial pylossless/qc.py, including the component topographies, scrolling ICA time courses, and raw-versus-cleaned comparison tested in pilot-qc. The pull request added 481 lines across seven files.

The merged code was still pretty rough. It even kept my hacky local file based approach while the interaction settled down. That was fine. I didn’t need the prototype to produce immaculate code that could be copied over unchanged. I needed proof that this collection of plots and interactions could work at all.

That changed the work completely. I wasn’t trying to make the full QC system while juggling six libraries in my head anymore. I just needed to take the important parts from what worked, and bring that over. I now had my old QC workflow back!

I’ve since gone back and cleaned up some chunks and improved the local file calling. But I couldn’t have done any of that without the throwaway prototype to fall back on.

Try the intimidating version somewhere disposable

A throwaway prototype is great when a feature feels too large because a bunch of unknowns are stuck together. Try pulling them out of the main project. Use real enough data to test the idea, but don’t spend the first day deciding where every class belongs. Nobody cares yet.

Make the smallest repository that can answer your question. Let it use a weird little file for communication if that gets the idea on screen. Let the commits say wow this is amazing. Clean up whatever survives later.

If your favourite project has a feature you’ve been avoiding because it looks like one big impossible blob, give it a disposable repository and an afternoon. See what happens.

The response also confirmed why I wanted to give the talk. People are constantly asking me agents and what they should be doing with them. They’re watching other researchers and developers move very quickly with these tools, and there’s a real fear of missing out. They want to try them, but they don’t necessarily know where to start or how much of the output they should trust.

My answer is to start with code you already trust, then put some AI slop around it.

The rest of this post are my thoughts about the various points I bring up during the talk.

The title was a bit of a lie

The title says you should write more code. This is technically true, but it depends on what we count as code.

Most researchers don’t need an agent generating more of the main scientific loop. That is the meat and potatoes of the project. It includes the cleaning rules, statistics, simulations, models, and transformations that can change the result. The researcher needs to understand and own those decisions.

A lot of useful code lives outside that loop:

• tests and synthetic examples
• input validators
• diagnostic plots and reports
• logging and debugging tools
• profiling scripts
• build and release automation
• documentation

People tend to skip this stuff. It takes time, some of it is boring, and it may require programming knowledge that isn’t directly related to the research question. It is also exactly the work that can make the main analysis easier to trust.

The title I really wanted was “Use AI agents to write slop to verify your code.” It probably wouldn’t have attracted 300 registrations.

I asked Claude to do my stats

I built a synthetic reaction-time study for the talk. It had 48 participant files and a 2x2 repeated-measures design. Each participant should have completed four conditions with 80 trials in each condition.

Of course, the data wasn’t actually that clean. One file duplicated an existing participant under a new acquisition ID. Two participants didn’t finish. Some had near-chance accuracy. Others had reaction times that were implausibly fast or slow.

All of these are normal research-data problems. They’re also serious enough that letting those files contribute to the final result would be wrong.

I gave the data to Claude three times.

For the first attempt, I used this prompt:

These CSV files are the results from my experiment. Do my stats for me and make a plot I can use in my paper.

This is a terrible prompt on purpose. Claude still produced a polished plot and a significant interaction with a p-value of 0.040. It looked like a finished analysis.

It turns out it had invented an accuracy exclusion, picked its own reaction-time summary, allowed comically large response times, and included incorrect trials. There were decisions all over the place that the researcher didn’t make and might not even notice.

The result was blatantly wrong, but it looked nice.

Never do this.

A better prompt still wasn’t enough

For the second attempt, I explained the study design. I described the factors, outcome, repeated-measures analysis, hypothesis, and how incorrect trials should be handled.

The code improved. Claude gave a more conservative p-value of 0.087. It also showed hints that it could produce useful checks around the analysis.

This is probably where a lot of people stop. The response knows the terminology, the figure looks good, and the result feels more defensible because the prompt was detailed.

There were still problems in the data that anyone familiar with the study would want to investigate. More context made the output better, but good prompting did not replace experience or domain knowledge.

This is also why I’m not especially interested in advice that boils down to “write a huge prompt.” Context is useful. It does not prove that the code handled the real files correctly.

Own the logic and delegate the annoying parts

For the third attempt, I supplied the core statistical code myself. It came from previous work, and I understood what it was doing. Claude wasn’t asked to choose the analysis.

Instead, I asked it to write the surrounding checks. Report any of the missing four conditions. Report files with too few trials, high error rates, or reaction times outside the plausible range. Check participant identities. Show me potential failures.

This found all eight problematic subjects, including the duplicate participant. After reviewing and handling those cases, the analysis used 40 participants and returned the expected interaction with a p-value of 0.037.

The third attempt was better because I owned more of the important code, not because Claude suddenly became more trustworthy. The agent took criteria I supplied and turned them into reports and checks. That gave me things I could inspect before accepting the result.

This is the kind of AI-generated slop I want more of.

If a validator is a bit ugly but catches a duplicate participant, great. If a throwaway plot makes a broken distribution obvious, it did its job. If an alternate implementation disagrees with my main calculation, I now have something worth investigating.

None of that code needs to become permanent infrastructure. Its job is to answer a question or provide evidence.

This works outside toy statistics

The same idea fits notebooks, shared pipelines, and SLURM jobs.

For a notebook, an agent can add data-shape checks, diagnostic cells, caching, or a small script that tests code pulled out of the notebook. For a shared pipeline, it can validate inputs, produce reports, or turn a previously discovered failure into a regression test. For a SLURM workflow, it can add preflight checks, logging, resource summaries, checkpoints, and checks that distinguish a complete output from a partial one.

The stakes change as the work gets larger. A bad notebook cell might waste an afternoon. A bad cluster job might wait in the queue for days, consume a large allocation, and leave partial files that look usable. Catching the same mistake before submission is much cheaper.

There are also obvious limits. Don’t upload data unless you’re allowed to. Run unfamiliar generated code in a sandbox, container, or virtual machine. Keep a human involved when mistakes can affect results, people, infrastructure, or budgets. You own the problems even when an agent wrote the code.

Try one check

Don’t start by giving an agent a new analysis and hoping it gets everything right.

Take a script, notebook, or pipeline that you’ve already used enough to understand. Something battle-tested. Ask the agent to add one check that doesn’t touch the meat and potatoes.

You could ask for a synthetic case where you know the answer. Ask it to verify that every participant has the expected conditions. Have it make an exclusion report, plot a suspicious distribution, or write a second version of a calculation so you can compare the outputs.

Review what it writes. Run it. See whether it tells you anything useful about code you already know.

That is a much safer way to learn what agents are good at, and it produces something useful even if the generated code is disposable. The point isn’t to trust AI more. The point is to cheaply make more of the stuff that helps you trust the important work.

This sounds simple, but it isn’t. A Python script can live almost anywhere, but a formal package needs the right structure to be built into a wheel. It needs dependency metadata, versions, releases, a licence, and some way to test that it still works.

Cookiecutter gave us a standard template. Instead of remembering every file and setting and creating them manually, we answered a few questions and started with a working package. I gave a talk about that workflow in 2022.

Personally, I stopped shipping tools that needed it, so I stopped thinking about Cookiecutter. Recently, I needed to package something again, and an obvious question came up. Why use a project template when I can ask an agent to build the project?

A small comparison

For comparison, I used the same throwaway project for each attempt. It’s a small package called csv-summary. All it does is read a CSV file, summarize its numeric columns, and provide a csv-summary command. I also wanted tests and enough infrastructure to install, check, version, and release the package.

I tried three different options:

1. cookiecutter-pypackage without an agent
2. An agent starting from an empty directory
3. The Cookiecutter project followed by the same agent

I did this with one prompt, not a benchmark. The goal was to see what each approach considered a finished package and where I would have to step in and fix things.

Cookiecutter by itself

Cookiecutter has changed a lot since my 2022 talk. The current template uses uv, Ruff, pytest, ty, and GitHub Actions. uv wasn’t even mainstream until late 2024. The template now includes workflows for continuous integration, documentation, CodeQL, and publishing to PyPI with trusted publishing. It also has Dependabot configuration, release tooling, security documentation, and a changelog.

This is a lot more infrastructure than this example needed, but that’s pretty much the trade-off for Cookiecutter. The opinionated set of files is great, but if the package is truly small, this is overkill.

The agent by itself

Starting from an empty directory, the agent produced the leanest project. It chose Polars, added a src layout, wrote two focused tests, configured Ruff and pytest, built both a wheel and source distribution, and checked them with Twine.

The agent also stopped at a fairly light definition of “done.” Releases used a documented sequence of uv, Hatch, and Twine commands. There was no continuous integration, automated PyPI publishing, documentation site, dependency updater, or security scanning.

This is pretty much what you’d want for a small internal tool. It’s less convincing as the default for a package that other people will depend on.

The prompt had asked for release infrastructure, but the agent gave me a workable manual process. Cookiecutter gave me a repeatable and automated one.

A better prompt could have asked for every missing piece. Of course, writing that prompt means knowing which pieces to request.

Cookiecutter followed by the agent

The combined approach produced the most complete result. Cookiecutter supplied the project policy, while the agent spent its time implementing the package. Its formatting, linting, type checks, tests, and package build all passed.

The agent went further with the combined example. It added a minimum version for pandas, locked the resolved dependencies, and discussed which Python versions to support.

Cookiecutter had already handled much of what my prompt requested. The agent had less setup work to do, so it spent more time on package-specific decisions than it did in the empty-directory test.

Templates still have a job

In general, an agent is good at doing the work in front of it. A maintained template is good at recording work that is easy to forget.

That difference is important. I can ask an agent to create CI, configure trusted publishing, pin GitHub Actions, add release notes, and set up documentation. I can also forget to ask for one of those things at any stage.

Cookiecutter records those decisions in files that have been used and tested together. With a template, the agent’s job is smaller. It doesn’t need to invent a release process or choose a documentation system. It can focus on other things, such as the consequences of a dependency or which Python versions to support.

There are concerns, of course. A Cookiecutter template can get out of date. A technology can fall out of style while the template keeps generating it. You could even argue that the whole thing could be a Markdown file or an agent skill. Still useful for a beginner, though!

What I would use

For a small internal script that only needs to be installable, I would just let an agent drive. There’s no need for all the extra GitHub features.

For a package that I plan to publish and maintain, I would start with Cookiecutter and then give the generated project to an agent. Cookiecutter would provide the boring, repeatable decisions. The agent would adapt them and run with it.

Interesting to see how far we have come in just a few years.