The Quiet Craft of Attention
What we lose when our models stop noticing the small things.
There is a thing that happens when a model gets very good at one task — it starts forgetting how to notice. This is a small essay about what it means for a system to pay attention, and why I think we've been measuring the wrong thing for a long time.
The trouble with metrics
Most of the metrics we use to evaluate vision models reward consistency. A good model produces the same answer twice. A great model produces the same answer ten times in a row. But consistency is not understanding — it's the absence of confusion, which is something different entirely.
The map is not the territory. And the metric is not the model.
When we stop optimizing for noticing and start optimizing for not-being-wrong, we end up with systems that are confidently bored.
What I'm trying instead
I've been spending the last few months on a small project that asks: what if we trained a model to be uncertain in interesting ways? Not uncertain in the way a calibrated classifier is uncertain — uncertain in the way a person is uncertain when they walk into a room and feel that something has changed but can't quite say what.
It's early. The numbers are not particularly good. But the model behaves differently, and that, I think, is worth chasing for a while.