Jake Wang

An AI-assisted PR is still your PR

Jake Wang8 min readMerged

阅读中文版

A shader that needed 11.5 GiB of memory to compile now needs 0.66 GiB, and the compiled output is byte-for-byte the same. The fix is 12 added lines and 2 removed, in SPIRV-Tools, the Khronos Group’s toolkit for the shader format Vulkan uses. It was merged twelve days after it was opened.

I typed four sentences before that pull request existed. A set of AI agents did the rest: found the bug, wrote the fix, wrote the test, measured it, and opened the PR under my name, 23 minutes after my first message.

The PR says the work was assisted by OpenAI Codex. This post is the long version of that sentence: what the agents did, where they were wrong, and what was left for me.

Why I was there

Sony keeps moving the PlayStation away from discs, and this summer it announced that more than 550 movies and shows people had bought would be removed from their PlayStation libraries. That got me interested in projects that let people keep running the PS5 games they own on a PC.

AnyPS5 is one of them. Its README describes it as a tool for automatically porting PS5 executables to Linux and Windows, with no emulation: it relinks the executable into the host’s native format and supplies its own implementations of the system libraries. Part of that job is recompiling the game’s shaders to SPIR-V, and it uses SPIRV-Tools to optimize them.

My work is observability and data infrastructure. I had never worked on a shader compiler.

Four sentences

Times are UTC, on October 12, 2026.

  • 03:30 I asked Codex whether AnyPS5 was open source.
  • 03:31 I asked it to find three to five issues I could contribute to.
  • 03:36 It told me the interesting ones needed hardware I did not have at hand. I asked: couldn’t you test in Docker?
  • 03:38 I told it to go ahead and open PRs, testing in Docker or however it could.

One of the issues it had shortlisted was a memory problem: with shader optimization on, the game Demon’s Souls grew to about 16.6 GiB in its first 26 seconds of start-up.

Then it worked by itself. A lead agent started three sub-agents: one to set up a Linux build in Docker, one to investigate the optimizer, and one that began on a different issue and was later reused as a reviewer.

  • 03:44 A generated shader of about 2 MB drove optimizer memory to roughly 4.7 GB. The agent’s first theory, that the input was being validated twice, did not explain it. Removing the duplicate validation left the peak where it was.
  • 03:47 It isolated the growth to one pass in SPIRV-Tools, redundancy elimination.
  • 03:50 The fix passed all 3,794 upstream optimizer tests, plus a new regression test written by the reviewer agent.
  • 03:53 The PR was open.

What was wrong

Redundancy elimination walks each function’s dominator tree and removes computations whose result is already available. To know what is available, every block gets a map from value numbers to the IDs that hold them. A block starts with its parent’s map and adds its own entries.

This is the loop that did the walking:

for (size_t next_node = 0; next_node < todo.size(); next_node++) {
  modified |= EliminateRedundanciesInBB(todo[next_node].node->bb_, vnTable,
                                        &todo[next_node].value_to_id_map);
  for (DominatorTreeNode* child : todo[next_node].node->children_) {
    todo.push_back({child, todo[next_node].value_to_id_map});
  }
}

todo is a work list that only grows. Each child is pushed with a full copy of its parent’s map, and nothing is removed after a block has been processed. Every map stays alive until the function returns.

On a long chain of blocks where each block adds a new value, that adds up:

block 0   map has 1 entry     kept
block 1   map has 2 entries   kept
block 2   map has 3 entries   kept
...
block n   map has n entries   kept

That is roughly n²/2 entries held at once, to process a function with n values in it.

The fix

const auto& children = todo[next_node].node->children_;
for (size_t child_index = 0; child_index < children.size(); ++child_index) {
  if (child_index + 1 == children.size()) {
    todo.push_back({children[child_index],
                    std::move(todo[next_node].value_to_id_map)});
  } else {
    todo.push_back(
        {children[child_index], todo[next_node].value_to_id_map});
  }
}
todo[next_node].value_to_id_map.clear();

The last child takes the parent’s map by move. Earlier children still get copies, because each sibling subtree has to keep its own state. Then the parent’s entry is cleared. The traversal order does not change. On a chain, there is one live map instead of n.

Peak memory of the pass on a generated chain of blocks
  • Before
  • After
4,096 blocks
Before
402 MiB
After
17 MiB
8,192 blocks
Before
1,572 MiB
After
30 MiB

Peak resident memory, medians of three fresh-process runs. macOS arm64, AppleClang 21, Release build. Source: the pull request description.

Linear blocks Peak RSS before Peak RSS after Time before Time after
4,096 402.47 MiB 16.92 MiB 0.17 s 0.02 s
8,192 1,572.44 MiB 29.88 MiB 0.62 s 0.06 s

Doubling the input multiplied the old memory use by about 3.9 and the new one by about 1.8.

The evidence the agents produced

I could not judge this fix by reading it. What I could judge was the evidence that came with it, and that is the part I would ask of any agent, and of any person.

  • A regression test aimed at how the fix could go wrong. The risk in moving a map is that a block loses values it should inherit, or that two siblings share state. The test builds a parent, two branches and a merge block, and checks both.
  • A mutation to prove the test can fail. The reviewer agent changed the fix to move the map into the first child instead of the last. The test failed.
  • Identical output. Old and new builds produce byte-identical modules, and both pass the validator. The change is supposed to affect memory only.
  • The same patch inside a real consumer. Applied to the SPIRV-Tools revision AnyPS5 pins, in Ubuntu 24.04 Docker with GCC 13.3, a generated shader through AnyPS5’s full optimizer wrapper went from 437.95 MiB to 41.70 MiB.
  • A plain statement of what was not shown. The PR says: “The reported game workload has not been reproduced, so its resolution remains unconfirmed.”

Where the agents were wrong

It said it could not test. Its first answer was that the good issues needed hardware and a platform I did not have set up. That was overstated, and one question from me was enough for it to say so: the project’s own CI runs on a CPU-based Vulkan renderer, so Docker was fine.

Its first theory was wrong. It spent several minutes on duplicate input validation. It dropped the theory because it measured, and the number did not move.

It did not run the project’s formatter. The only thing a maintainer asked for in review was “Please try to fix the formatting.”

It could not sign the contributor agreement. That needs a person.

What was left for me

Five things, none of them code.

  1. Choosing where to point it. The project and the reason were mine.
  2. Not accepting “can’t”. The Docker question.
  3. Asking for real evidence. The next day I asked for a benchmark on shaders from an actual title, not a generated one. The agents found a publicly released homebrew game, MedievalTennis, and measured its embedded shaders: the highest compile peak went from 695.80 MiB to 94.84 MiB, with every successful output matching byte for byte. The raw records are in a gist.
  4. Getting the fix to users without waiting. I asked whether AnyPS5 could carry the patch directly. That became a second PR, merged a day later.
  5. Understanding what I had signed. The PR was already open when I sat down and had the agent explain it to me: what the pass does, what the map is for, why the old code kept every copy. I should be able to defend anything that goes out under my name, and by the time a maintainer looked at it, I could.

The people and agents on the other side

The bug report itself opens with the line “Posted by an AI agent on behalf of Tyler.”

A contributor I had not worked with replayed a shader captured from Astro Bot on Windows 11. Peak private memory went from 11.516 GiB to 0.663 GiB, with identical output. That is the number at the top of this post. Their comment says their testing was assisted by Codex too.

The day after the downstream patch merged, the original reporter came back. On the saved Demon’s Souls shader modules, the largest one had run out of memory at more than 16,270 MiB before the patch. With it, the same module finished at 2,796 MiB. They had not rerun the game itself, and said so.

So an agent filed the bug, agents fixed it, and agents measured the fix on three people’s machines. The humans in the loop were the three of us who pointed them, and the maintainers who approved and merged it.

Review

A SPIRV-Tools maintainer approved the PR two days in and asked for the formatting fix. CI was noisier than review: two macOS jobs died downloading a build tool, and two downstream smoke tests failed with the same failures as the unmodified base branch. The agent wrote that up and noted it was not a controlled comparison. A maintainer replied “You can ignore the smoketests,” restarted the other two jobs, and the PR merged that day.

Three things I would tell someone sending an AI-assisted PR

  1. The less you know the code, the more evidence you owe. The agents made it possible for me to change a compiler pass. They did not make me someone whose word a maintainer should take.
  2. Write down where your evidence stops. “Not reproduced” and “not a controlled comparison” cost nothing to type, and they are what a reviewer checks first.
  3. Say that you used AI. The maintainers are going to wonder anyway.

The fix was twelve lines, and I did not write them. The PR still has my name on it. That part does not delegate.

Sources