nuclear-news

Automation in nuclear weapon systems: lessons from the man who saved the world

, Medium.com. NinaMiller, 16 July, 21, In 1983, the world came within a phone call of nuclear annihilation. When an alert of incoming ballistic missiles registered at an early warning command centre outside of Moscow, Lieutenant Colonel Stanislav Petrov had to decide whether or not to confirm the signal to his superior, an action which could have sparked a catastrophic nuclear exchange. Rather than escalate the report up to Soviet leadership, Petrov — who felt he ‘was sitting on a hot frying pan’ — decided the missile alert was a system malfunction.

Later called ‘the man who saved the world’, Petrov demonstrated an astute understanding of the limits of machine analysis. What can modern policymakers learn from Petrov’s experience about the impact of automation on accidents in nuclear weapon systems?Available information indicates that US officials are integrating greater amounts of automation and potentially machine learning in nuclear command, control, and communications (NC3). Indeed, the fields in which increased automation is being considered vary from predictive maintenance and data analytics to cybersecurity, tracking of adversary submarines, and early warning systems. Human operators often ‘overtrust’ automated systems in other high-consequence environments like civil aviation and medicine, yet it remains unclear exactly how automation misuse could increase the risk of nuclear accidents or escalation………..

an inherent paradox when it comes to human — machine interaction: the more reliable and useful an automated system is, the less likely human operators are to critically assess and pay attention to its function. In other words, the probability of a catastrophic mistake caused by automation bias or complacency in NC3 will be highest for consistent, highly reliable systems with a high level of automation

Will the next Petrov make the right decision? To decrease the risk of automation misuse and instability, next generation command and control will need to reward vigilance, give operators the time and ability to consult additional information, and ensure that nuclear postures in the United States and elsewhere do not encourage over-reliance on machines in a crisis.

Decision-support systems that develop recommendations for human operators about the use of nuclear weapons are likely to involve the highest risk of automation misuse. Machine advice could be misinterpreted or uncritically trusted when the systems perform well in peacetime and wargames, leading users to develop a ‘learned carelessness’ when using the system. The lumberjack effect is perhaps the most counterintuitive and dangerous paradox — if the Soviet early warning system had been highly reliable and vetted, Petrov might not have hesitated.

As US officials contemplate the proper role of machine learning in a modernized NC3 infrastructure, they should be careful not to take the wrong lessons from Petrov’s experience. Human supervision is not enough. Healthy human-machine teams need opportunities to train together and learn from mistakes, which is difficult or impossible for certain NC3 functions like early warning or force planning. Proposed solutions like explainable AI and enhancing trust in AI could actually be counterproductive if they create false expectations of machine reliability or inadvertently encourage complacency. Nuclear modernization in the United States and elsewhere should take as a starting point that the paradoxes of automation cannot be solved, only mitigated and managed.

Nina Miller is a PhD student in MIT’s Department of Political Science and currently a Research Associate with Lawrence Livermore National Laboratory’s Center for Global Security Research (CGSR). Her research focuses on the intersection of international security, political psychology, and technology.   https://medium.com/international-affairs-blog/automation-in-nuclear-weapon-systems-lessons-from-the-man-who-saved-the-world-d39aa2f4da5a

July 17, 2021 - Posted by Christina Macpherson | 2 WORLD, weapons and war