Timezone: »

System Safety Engineering for Social and Ethical ML Risks: A Case Study
Edgar Jatho · Logan Mailloux · Shalaleh Rismani · Eugene Williams · Joshua Kroll

Governments, industry, and academia have undertaken efforts to identify and mitigate harms in ML-driven systems, with a particular focus on social and ethical risks of ML components in complex sociotechnical systems. However, existing approaches are largely disjointed, ad-hoc and of unknown effectiveness. Systems safety engineering is a well established discipline with a track record of identifying and managing risks in many complex sociotechnical domains. We adopt the natural hypothesis that tools from this domain could serve to enhance risk analyses of ML in its context of use. To test this hypothesis, we apply a best of breed'' systems safety analysis, Systems Theoretic Process Analysis (STPA), to a specific high-consequence system with an important ML-driven component, namely the Prescription Drug Monitoring Programs (PDMPs) operated by many US States, several of which rely on an ML-derived risk score. We focus in particular on how this analysis can extend to identifying social and ethical risks and developing concrete design-level controls to mitigate them.

#### Author Information

##### Edgar Jatho (Naval Postgraduate School)

Commander Jatho is a member of the Permanent Military Professor Community. He is currently pursuing a PhD in Computer Science at the Naval Postgraduate School in Monterey where his research specializes in trustworthy artificial intelligence and developing general approaches to eliminate or mitigate the problem of adversarial examples in deep neural network classification systems. Prior to May 2020, CDR Jatho served in the Cryptologic Warfare community. Previous tours included Navy Cyber Defense Operations Command as the N9 Defensive Cyber Operations Afloat Department Head, CARRIER STRIKE GROUP TEN as Cryptologic Resource Coordinator, National Security Agency as Special Access Program Central Office and Special Technical Operations Deputy Chief. His awards include two Defense Meritorious Service Medals and two Navy Meritorious Service Medals.