Troy Kelly

6.2 is a problem for OpenAI

OpenAI's models are causing slowdowns and roadblocks for my current development pipeline - I explore my experience verse Anthropic models.

WritingDevelopmentOperations2 min read

In the last few weeks, I've noticed my agent supervision was no longer the occasional intervention to get things back on the rails, or to reset the fixations that agents have tended to get on drastically overengineering a single item while not shipping anything else.

I was doing a lot more babysitting. My eyes had to stay on the harness, on the work of the agent - because it just kept stopping. I hadn't seen this sort of silent work refusal for a while, all the frontier and open models of decent size have been pretty good, especially in DeepSeek Harness, at working for hours and orchestrating operable code, tests, etc.

The problem model was Sol 6.2, so I tried moving that work to Astra 6.2. Anecdotally, the problem got way worse with the better equipped model.

The quality of work was not degraded. It wasn't "blow my mind" impressive - but it wasn't bad by any measure. But there was just so little of it, and every time I'd get the agent back to work, it would find any possible excuse. The reasons for the refusal ranged from sort-of-legit but still obtuse repository rule interpretation through to just stopping, often while even saying something along the lines of "this work is still not shipped...".

I tried codex cli and other harness' - but there was no appreciable difference. Agents would do a little bit of work, then stop. There were a few times that work continued - but both Sol and Astra tended to happily keep working only when they were hyper focused on administrivia, from CI to tests through to documentation - anything other than actually shipping code.

The sessions that would run for hours, consumed hundreds of millions of (total) tokens - and often changed only a few lines of documentation, or created detailed ADR's - but never moved to actual shipping.

What's interesting is that I think Opus 5.5 and Fable 5.2 know how to better prompt both Open AI 6.2 models - because my workflow typically involves an architectural panel, adversarial review, tests, code, adversarial and security review, pr/merge.

I'm seeing that Opus and Fable are both calling on OpenAI's models for the adversarial reviews, and that those reviews are coming back really high quality. The reviews catch esoteric issues through to security problems - so from what I experience - the OpenAI's models are quite powerful in this space.

Where they are lacking, at least for me, is that they just can't be the coders with their virtual fingers on their virtual keyboards.