Software Safety
By 
If you supervise people moving quantities of soil and debris with big machines, or people doing things with plutonium in glove boxes, questions of safety quickly become important.
Software developers have been much less aware of safety issues. What can go wrong besides lacking an ergonomic chair?
Social media sites and “AI” development are beginning to highlight safety in software. I’d like to throw out a simple principle that the software people might like to think about.
There are three levels of safety practices:
- Engineered safety. This is safety built into a machine. Belt guards to prevent finger amputations. Those annoying backup beeps that tell you a large machine may be headed your way.
- Administrative safety. The regulations that Supreme Court is tearing down. Posted limits on how much plutonium may be in a glove box.
- Individual awareness. Why you have to take that training. Do not do stupid stuff, with the stupid stuff spelled out.
Historically, software safety has been predominantly at the third level. Choose a password; the specifications imposed on you are administrative safety, imposed after we found out that “password” and “1234” were faves. Don’t share various sorts of personal data.
Structures of apps like Facebook have been designed in opposition to safety precepts at the first level, being intended to harvest people’s personal data and manipulate it so as to manipulate them.
The software industry has consistently fought safety at level 2, in the form of regulation.
If Los Alamos’s plutonium facility were run the way the software industry is, it would long ago have had a major criticality incident that would have burned it and half of Los Alamos to the ground, killing scores of people. They’ve made their mistakes, but they haven’t actively tried to undermine basic safety standards.
Cross-posted to Nuclear Diner