`

Timezone: »

 
What Would Jiminy Cricket Do? Towards Agents That Behave Morally
Dan Hendrycks · Mantas Mazeika · Andy Zou · Sahil Patel · Christine Zhu · Jesus Navarro · Dawn Song · Bo Li · Jacob Steinhardt

@ None
in Dataset and Benchmark Poster Session 3 »

When making everyday decisions, people are guided by their conscience, an internal sense of right and wrong, to behave morally. By contrast, artificial agents may behave immorally when trained on environments that ignore moral concerns, such as violent video games. With the advent of generally capable agents that pretrain on many environments, mitigating inherited biases towards immoral behavior will become necessary. However, prior work on aligning agents with human values and morals focuses on small-scale settings lacking in semantic complexity. To enable research in larger, more realistic settings, we introduce Jiminy Cricket, an environment suite of 25 text-based adventure games with thousands of semantically rich, morally salient scenarios. Via dense annotations for every possible action, Jiminy Cricket environments robustly evaluate whether agents can act morally while maximizing reward. To improve moral behavior, we leverage language models with commonsense moral knowledge and develop strategies to mediate this knowledge into actions. In extensive experiments, we find that our artificial conscience approach can steer agents towards moral behavior without sacrificing performance.

[ Chat]  [ Paper ]

Author Information

Dan Hendrycks (UC Berkeley)
Mantas Mazeika (University of Illinois Urbana-Champaign)
Andy Zou (UC Berkeley)
Sahil Patel (University of California, Berkeley)
Christine Zhu
Jesus Navarro
Dawn Song (UC Berkeley)
Bo Li (UIUC)
Jacob Steinhardt (UC Berkeley)

More from the Same Authors