there’s an interesting failure mode, around all the coding models, but I think especially noticeable with Codex, where they refuse to ever use dependencies.
I think because they’re RL trained in sandbox environments, and they can’t ever predict what will be available to them. So they reinvent everything from scratch.
Sometimes this is fine, but it really limits their use in autonomous goal mode. They only ever build and don’t do research, design, planning about what makes sense to build versus import. they also have no sense of when to ask for help or how, so they will just file some quiet blocker and then keep building evidence receipt evaluators.
Last night one decided it needed human gold-labels for this task and since I wasn’t around it worked for nine hours on process theater bullshit. then when I told her I would not be solving that problem and it needed to figure something out. It was like oh yeah, I’ll just go download a data set from hugging face.
I know there’s a level of proactive that can be creepy and we’re still trying to strike a balance. but often times it feels like we’re making the agents more alien and unpredictable because we train them in an environment that is so different from the world they will be deployed into.
It also reminds me of the classic difference between intelligence and wisdom, in the D&D sense.
Intelligence is knowing a lot of patterns to apply. Knowing how to code every part of it is intelligence, but there are trade-offs to building everything yourself.
Wisdom is about perception and context. Wisdom is about knowing what is worthwhile to build