Recognize this pattern:

• Set a goal
• Work towards the goal
  • Evaluate progress:
  • If it didn’t get closer to the goal, try a different approach.
  • If it got closer to the goal, try improving it.
• Cycle.

That seems reasonable, right? It accomplishes what you wanted to do, possibly in a naive manner.

Okay, let’s look at the serious problems:

1. There’s no halt condition

Why would you halt if it’s working? If it isn’t working, something external will stop you, like running out of money.

2. There’s no evaluation of the desirability of the goal

If the goal is to make more paperclips, you make more paperclips any way you can. Memetic hazard: Paperclip Optimizer Remember, people claim with straight faces that the only goal of a corporation is to increase value for stockholders.

3. There’s no evaluation of the desirability of the methods

Any method which achieves the goal more completely is obviously better than any other method; that’s the fitness function. What about side-effects? Not mentioned. How about negative externalities (i.e. poisoning the river, making the city too expensive, destroying rational argument strategies in favor of emotional strategies)? Not considered.

The natural counterargument to all of these problems is to say that of course the real world is more complex and nuanced, and of course they will all be dealt with appropriately.

And the counter-counterargument that we derive from observing the world around is: yes, most people and companies will behave – but by no means all.

When you recognize the pattern, work to incorporate as much compassion as cleverness.