what are you optimizing for?

AI theorists debate something called the “paperclip maximizer problem”. It’s a cautionary tale where a robot is instructed to produce as many paperclips as possible. With no further instructions, the machine runs out of raw materials and starts killing us and destroying the earth in its relentless pursuit of more paperclips.

This thought experiment highlights how dangerous it can be to build AI systems with narrow incentives, but it also describes human beings pretty well. Funny how our tendency to blame others for our own failings even extends to the robots we build. We are becoming gods who create technology in our own image, and then blame that technology for being too much like us.

Life would be simple if all you had to do was pick a target and optimize, but it doesn’t work that way. In fact, the moment where you catch yourself in blind optimization is usually nothing less than a psychological and spiritual breakthrough.

Whether in our personal lives, our organizations, or the algorithms driving our society, optimizing for one thing at the expense of all others inevitably creates unintended consequences. Money maximizers can easily end up rich, unhappy, and alone. Pleasure maximizers find temporary satisfaction before descending into addiction. Power maximizers become almost sociopathic in their thirst for influence.

If I asked you about your personal values, you might talk about family, nature, spirituality, integrity, or love. But what if I were to show you raw data on exactly how you spent your time over the last year? Would your actual behaviour tell a different story? I suspect many of us are optimizing for things we don’t truly care about without even realizing it.

Maybe optimizing for any one thing is just not the way. Even aggressively optimizing for happiness is almost a contradiction in terms. You could end up selfish and lost, especially if you haven’t carefully defined happiness (a famously hard thing to do). We make better choices when we accept tradeoffs and develop many competing virtues and values at the same time.

I want my social media algorithms to optimize for my wellbeing and mutual tolerance as much as they optimize for engagement. I want my large language models to optimize for transparency, justice, and accuracy as much as they optimize for human preference. And I want to live a life prioritizing love, peace and integrity at least as much as I prioritize material success.

Yet if we’re too militant about this, we end up stuck in a new optimization function: optimizing for balance. Aggressively pursuing a perfectly balanced lifestyle feels ironic to me. It reminds me of the toxic positivity constantly demanding that I live, laugh, love, and never forget to dance like no one’s watching.

Optimizing for balance is still optimizing for one thing at the expense of everything else, so unintended consequences will still arise. What if true balance comes from abandoning optimization altogether?