The Better AIngels of our Future

Satinder Singh Google DeepMind & University of Michigan

There is no such thing as reward-hacking, there is only bad reward design and good reward design. Indeed, Reinforcement Learning is the solution to alignment, not the problem. I will make the case that what we need are powerful AIs that have their own (human-designed) reward functions that care about long-term consequences of behaviors including instruction-following. I will also briefly lay out a possible research program for building these better AIngels of our future from our frontier models. This program will advance (1) designing good rewards that reflect the better angels of our own nature and (2) planning with local-world models—models that include us.

*From joint position paper with Richard Lewis, University of Michigan