Remote NIF Control Software Updates Lead to Less Stress, More Efficiency
February 11, 2021
First of three articles describing NIF’s rapid restart following the 2020 pandemic shutdown.
Updates to the National Ignition Facility’s Integrated Computer Control System (ICCS) software happen quarterly. The procedure with each upgrade was pretty standard—deployment of the new software, regression testing, and then a test shot, all performed under the pressure of knowing a single mistake could take down the entire facility.
Until 2020, that is. When much of LLNL’s workforce began working remotely due to the COVID-19 pandemic, that included the NIF software team. Rather than becoming a negative disruption, this change led to improvements in the efficiency, safety, and overall experience for everyone involved.
For example, instead of team members having to crowd into the NIF Control Room for a long and uncomfortable day, they now work from the comfort of their offices, with their tools and resources readily available.
“The pandemic meant we had to rethink the entire process,” said Gordon Brunton, NIF&PS Control Systems Lead. “We had to get creative to determine how to accomplish the same work without a physical presence”
Pre-pandemic, these updates were hectic, frenzied affairs. The software team would pack the Control Room, pairing with an operational staff member at each terminal, and work through a comprehensive suite of test procedures to requalify the changes to allow shot operations to resume.
Since only operational staff are qualified to operate the laser, each qualification step requires verification by both the operator and software team member. This meant operators would sit at their terminals with a software engineer awkwardly leaning over their shoulders for hours observing the screen and software behavior.
“That set up was a no go with COVID-19 distancing protocols,” said Brunton.
In response, Mikhail Fedorov, NIF ICCS Software lead, and Suzy Townsend, NIF Software Quality and Standards lead, established new procedures that not only meet the COVID-19 safety protocols but also represent a substantial improvement in efficiency and safety. The software team recently completed its third successful remote software update since the pandemic began.
“We aren’t going back to the old way of doing things,” Fedorov said.
The team quickly adapted. For example, assigning each operational staff member with an smartphone replaced the need for physical pairing—the phone created a secure one-to-one audio/video communication channel that provides the remote software team member the necessary situational awareness without having to stand in the same room.
In addition, a previously established a remote support capability allowed any operational console to be viewed securely through videoconferencing. With this capability, remote diagnostic scientists can validate alignment without being physically present. The software team leveraged this capability by establishing a single 24-hour conference call coordinating all activities and monitoring testing status, which meant team members could work in their own offices, whether on site or at home.
“Reducing the personnel density and noise through the established new protocols has made it much easier to focus and has resulted in more effective and efficient qualification testing,” said Brunton,
To keep the dispersed team in constant communication, Fedorov hosts the software update like a sports broadcast, giving a play-by-play as they move through their tasks. He also plays the role of an air traffic controller, managing multiple tests that could cause conflicts and mitigating downstream consequences of delays and problems.
At the same time, Fedorov uses an internal communications application to update every action and status of each qualification step. These two remote communication mechanisms allow the team to constantly keep in touch and aware of issues requiring resolution.
“Many people listen in, including facility people,” Fedorov said. We never had this before, a wide, common understanding of what is happening step by step.”
The team found they can execute regression testing much more efficiently.
“With the internal application, everyone can see the common status. If an issue comes up, it gets resolved much more quickly,” Brunton said. “There is a lot of knowledge on this team, but we are only one-deep in some specific areas. We now can escalate much more quickly, solve problems, and keep a record of what happened.”
With all members of the software team working from their own workstations, they have access to software developer tools, notes, and other resources that would not be available in the control room. This has drastically improved the effectiveness of the support and, in turn, the efficiency.
The lack of socialization is a common issue for telecommuters, but in the case of a software update, this is a benefit. “Conversations are briefer than when we were face-to-face and issues are resolved more quickly,” said Brunton.
Townsend now observes the progression of shots with the shot director and lead operator from her workstation, giving her the opportunity for microbreaks that make the long hours more bearable.
“I don’t get as exhausted,” she said. “Sometimes after a long, stressful day we’d worry about people driving home, especially if the update hadn’t gone well.”
Another benefit has been with training new members of the team. “We can have a new member watch remotely as someone else performs operations,” said Brunton. “Before, we’d do that looking over someone’s shoulder. Now our new members have more opportunity to observe.”
This improved efficiency is here to stay. “The COVID-19 safety controls forced us to question why we did things a certain way. Even when we are back on site, I fully expect to continue doing software updates remotely because it is such an improvement,” said Brunton.
Next in This Series:
Part 2: “COVID-19 Protocols Keep NIF on Track” NIF & Photon Science News, February 25, 2021
Part 3: “Target Fabrication Team Helped Speed NIF’s Rapid Restart” NIF & Photon Science News, March 15, 2021
More Information:
“Keeping NIF's Symphony of Controls in Tune” NIF & Photon Science News, February 12, 2020
—Patricia Koning
Follow us on Twitter: @lasers_llnl
