Efficiency Improvements - 2015
July
Electronic Checklists Keep NIF Running Smoothly
Target Fabrication Steps Up to the Challenges
Every NIF experiment needs a target. That might seem obvious, but it’s far from routine. Fabricating targets for NIF is a multi-faceted process requiring constant adjustments to meet the changing demands of experimenters and to deal with new engineering and material science issues that have a way of cropping up unexpectedly.
Annual production of targets is predicted to grow from 380 now to 480 by the end of Fiscal Year 2016.
And despite the challenges involved in designing, manufacturing, and testing a constant stream of tiny, custom-made, precision-engineered targets, the NIF & Photon Science Target Fabrication Team, in partnership with colleagues at General Atomics (GA) in San Diego, has been able to keep up with NIF’s steadily increasing shot rate (see “NIF Lasers Continue to Fire at a Record Rate”) through several efficiency improvements of its own.
Following the example of the automobile industry and many other mass-production facilities, the LLNL Target Fab team is turning to robotics to automate such time-consuming processes as installing the ultrathin membranes called “tents” that suspend target capsules inside NIF hohlraums, and mounting the hohlraums in the cryogenic target positioners that hold them in the center of the Target Chamber. The team also has been using a precision robotic assembly machine to automate the assembly of NIF hohlraums since experiments began in 2009.
Robot Tenting Machine (RTM) operator Cres Alday observes as the RTM applies tents to NIF targets in the Bldg. 381 target fabrication facility. The robot has improved the consistency and quality of the targets and reduced technician training time for applying tents from three months for manual application to two weeks. Credit: James Pryatel “The tent robot saves us about four hours a target,” said NIF target development and fabrication manager Alex Hamza. “It takes us (an average of) 70 hours to build a target, so that’s about a five percent savings. Maybe five percent doesn’t sound like much, but when you’re talking 400 targets (a year), that’s another 20 targets, which is another 20 shots.
“The hohlraum insertion also saves us about four hours a target.” Hamza said. “But the biggest thing is (a new) automatic proofing station” used to test cryogenically layered deuterium-tritium and tritium-hydrogen-deuterium targets before sending them to NIF. “We’re still running it through its paces and commissioning it, but when that comes on-line it should save us an additional eight hours per target, a more than 10 percent savings—or another 40 targets.”
Now being commissioned, the auto target proofing station (ATPS) will automatically perform ambient and cryogenic temperature leak tests and electrical conductivity tests on fabricated targets and should reduce the time to perform these tests by one day, or about 10 percent of the total time to fabricate and test a new target. Shown with the ATPS during commissioning are (left to right) Steve Andrews, lead product development technician; Rob Acree, production support; Albert Wang, lead engineer; and Rizalde Marquez, target proofing operator. Credit: James Pryatel “When applied judiciously, bringing robotics into the picture can be very beneficial,” said NIF Target Fabrication program manager Abbas Nikroo, who joined LLNL last month after leading the target fabrication effort at GA for the last six years. Nikroo said GA engineer Lane Carlson has led the automation of a number of assembly and other processes to speed the fabrication of capsules and other target components.
GA uses phase-shifting diffraction interferometer (PSDI) technology to make digital 3-D holographic images that allow “spheremapping” the exterior surface features of target capsules with nanometer (billionths of a meter)-scale precision. “That’s a process that an operator could do maybe six to eight times a day a number of years ago,” Nikroo said. “By simplifying the procedure, GA got it up to about 20 per day with a trained operator. With the robotic system they can do up to 120 with the robot running overnight. So that was a big bottleneck that was solved.”
Ignition capsules also go through a process similar to that used in Lasik eye surgery to remove tiny mounds, or “pimples,” on the capsule shell. “If there are any pimples on the shell that can”t be polished, ahead of the polishing they cut down those pimples so that in the polishing process it can be completely planarized,” Nikroo said. “That was a manual effort—someone had to go find the pimple, position it, take it to the laser, do the operation.
“Now it’s all on a robotic system where the shell can be moved around and mapped, and then the 4pi Laser-Polishing Station automatically determines if there are mounds on the shell that are too high. Based on the measurements the system aligns it, moves the microscope head out of the way and brings the laser in, cuts down the mound, and then brings the microscope back in, measures and confirms that it’s been trimmed, and then moves on. So now the operator is basically out of the loop.”
These and other efficiencies are predicted to reduce the average fabrication time from 70 hours per target to about 55 hours by the end of this year. “At the beginning of Fiscal Year 2014 (October 2013) we (produced) about three and a half targets per week,” Hamza said. “When we’re done we expect to be at five and a half targets a week by Christmas of this year.” Annual production of targets for the wide variety of experiments conducted at NIF is predicted to grow from 380 now to 480 by the end of FY16.
Along with maintaining a high production rate, the Target Fabrication Team also must find ways to address several vexing issues that have arisen during NIF’s campaign to achieve ignition (see “Climbing the Mountain of Fusion Ignition”). “Currently there’s a big push to eliminate the tent,” which is thought to cause instabilities in NIF implosions, Hamza said. “Figuring out how you can position the capsule to within ten microns in the center of the hohlraum without the thin polymer membrane supporting it is quite a challenge.” 
A NIF hohlraum with the fuel capsule suspended in the tent. The fill tube is used to insert the fusion fuel in the capsule.Ideas include suspending the capsule with spider silk, or levitating it with magnets. “Levitation with magnets is very promising,” he said, “but you’ll have to add some kind of a magnetic material to the targets, which may or may not be OK. I don’t know how thick the layer has to be to have the magnetic properties that you need.”
Other challenges include fabricating depleted uranium hohlraums that are sturdy enough to use in experiments without the gold liners that normally support them; adding a silicon dopant to high-density carbon (diamond) capsules without creating excessive levels of silicon carbide; and avoiding non-uniform oxygen uptake in the plastic capsules used in the majority of current ignition experiments.
“We just learned in the last year or so that any kind of light in an oxygen environment will create oxidation of the plastic,” Hamza said. “If (the capsule) would be taking up oxygen uniformly, that’d be one issue and we’d be worried about it, but it wouldn’t be so bad. We don’t know that this is true, but if it would be taking up oxygen azimuthally—non-uniformly—that could create a seed for instabilities.”
NIF targets must be built to extremely rigid specifications (see “Targets”), but if they were all the same or similar, producing more than 400 a year would be relatively straightforward. But the team is asked to create 40 to 50 different types of targets, many consisting of entirely new designs with new serial numbers that are used in only a handful of experiments. “Experiments are one-off or a couple-off, and then you change the design a little bit based on the results of the experiment, and the serial number starts over again,” Hamza said. “We (currently) produce 380 targets and at least 160 of them are (new) serial numbers” with unique requirements.
A sampling of NIF targets. The target fabrication process starts with the physicists, who establish the target physics requirements for inertial confinement fusion, high energy density science, and Discovery Science experiments on NIF. Target science & technology experts and material scientists from LLNL and GA work closely with Livermore engineers to design the target, and the specifications are then provided to the GA technicians who fabricate the components. The final stage is assembly of the components in a 3,000-square-foot “class 100” cleanroom in LLNL’s Bldg. 381.
GA has held the contract to produce targets for LLNL and the other National Nuclear Security Administration laboratories since 1991. “The relationship between GA and LLNL has been beneficial, it’s been very close, and it’s been a good partnership,” Nikroo said. “In various areas it’s very much seamless, where you could think of the GA operation as an extension of what Livermore is doing. Over the years we’ve tried to keep it away from becoming a supplier-and-vendor type relationship because understanding what is needed, what’s the experiment, why the part is being made this way, is just as crucial as getting the drawing—it’s important that that’s part of the communication.”
Nikroo noted that while some target components are “off-the-shelf” items from other sources, GA, like LLNL, produces many specialty items—“components and parts that you just can’t get elsewhere.” GA also puts “a lot of effort into internal R&D investments that look for targets of the future,” he said. “That’s something I’m hoping to foster even more—having them do some of the R&D that we may not have enough resources and funding to do at Livermore.”
Recently Schafer Corp. of Arlington, Virginia, has re-engaged in target fabrication for the NNSA effort, including work on NIF targets through on-site assembly support at LLNL as well as fabrication of components at its facility in Livermore. Schafer brings additional capabilities which will be utilized as target demand for NIF experiments continues to grow in the coming years.
Electronic Checklists Keep NIF Running Smoothly
Getting NIF ready to move from one shot to the next within a few hours is a complex ballet involving an average of 100 separate tasks each day. Spent targets are removed from the Target Chamber, new targets are installed, target diagnostics are swapped out and in, large optics are exchanged, equipment is aligned—all while facility support systems are being maintained for peak performance.
For the first two years of NIF operations, these jobs and many others were choreographed using paper checklists that remind workers of the most important steps required and the required additional manual processing afterward.
But in 2011 the NIF Final Optical Systems Operations group decided to go paperless, developing a suite of electronic databases to integrate the tasks involved in exchanging and recycling final optics following NIF experiments.The tool has increased NIF’s efficiency by an average of 30 minutes per task or 50 task-hours per day.
“We were doing transactions for the Loop (optics recycling) process,” said Simon Cohen, Lasers, Deputy for Operations & Alignment and Optical Systems Manager, “and the process had inefficiencies. We were printing up procedures on paper, and handwriting things afterwards, and I just saw that there were many ways to improve on our process and bring us into the 21st Century—no longer having to print procedures and handwrite steps.”
Cohen spearheaded the effort to automate the scheduling, verification, automated completion of work orders, and tracking of step completion time of NIF’s optics transactions using FileMaker Pro database software. The new process enabled field technicians to use iPads instead of paper checklists to complete their work, take photos on the iPads that could be uploaded into the system, and use handheld scanners to ensure that the correct parts and equipment were on hand when needed for the jobs. The status of each checklist could be viewed from any computer in real time as the work progressed and metrics could be graphed for assessing areas of improvement.
A Target Area Operations technician uses an iPad to complete a continuous phase plate optic exchange. “This was December 2011,” Cohen said, “and I realized very quickly when I was implementing this that it had generic functionality—that you could pursue any type of procedure with step processes very efficiently. Management saw the success of this new process and provided IT manpower to port it into a NIF IT product.”
An IT team headed by Vamsee Lakmansani and then by Scott Reisdorf with Dan Potter went to work and implemented Cohen’s database in an Oracle application named NIF Electronic Operations—NEO for short. “It was a huge amount of work, because these things are buggy at first, and people are often reluctant to change,” Cohen said. “There’s a learning curve, and getting everyone to buy into it is a challenge. Once we had overcome the transition challenges from paper to electronic processes in FileMaker Pro we needed to repeat this same challenge, and get the NEO software fluid enough so that it worked as well as the existing one did in FileMaker.
“As part of our checklist implementation in NEO,” he said, “one important task I pursued with the TAO (Target Area Operations) techs—and most teams have done as they deployed NEO checklists—was a detailed review of each procedure with the intent to minimize steps and improve process efficiency.”
NEO was launched in August of 2012, and its use has grown steadily—both within NIF and among other Laboratory groups that use checklists. NIF groups currently using NEO include TAO, Target Diagnostic Systems, Cryogenic Operations, Beamline Systems, Classified Operations, Mission Support, Pulsed Power Systems, and Alignment and Optical Systems.
TAO engineering technician Mike Morris said he and his co-workers use NEO “to access daily work permits, radiological permits, daily assignments, and the checklists we need to do our jobs. And as soon as we finish our jobs we’re able to sync and update the information with the current configuration of the facility within five to 15 minutes. NEO is an invaluable tool that enables us to accurately and efficiently do our jobs in a much more productive way.”
Morris credited Cohen with working hard to overcome the initial resistance to the new system. “Change is hard,” he said, “and Simon was very diligent in implementing the system. He’s the reason this got off the ground. His persistence has paid dividends in helping us do our job as technicians.”
As of June, more than 6,500 checklists had been completed in NEO involving total worker time of about 500 days. The tool has increased NIF’s efficiency by an average of 30 minutes per task or 50 task-hours per day, helping NIF reduce the turnaround time between shots and substantially increase annual shot rates (see “An Increased Shot Rate at the National Ignition Facility,” Science & Technology Review, March 2015).
NEO Team members (from left): Scott Reisdorf and Dan Potter, NEO developers; Michelle Oliveira, NEO administrator; Allan Casey, Shot Data Systems section leader; and Simon Cohen, NEO conceptual designer. Not shown: Vamsee Lakamsani, Jackie Meeker, Elizabeth Palma, and Casey Shulz. Credit: James Pryatel “The beauty of NEO,” said NEO Administrator Michelle Oliveira, “was that as the user base increased, we used feedback to continuously improve and shape the product into something multiple groups could adopt.” Incorporating comments and suggestions from users has resulted in more than 20 updates since NEO was launched, including such enhancements as the ability to add voice notes to checklists and to send email notifications to system owners when problems are identified and checklists are completed.
“With this increased communication, something that’s happening in the field can quickly get translated back to the engineering side of the house, so they can make changes or get back to the technicians (to clarify) what’s going on,” Oliveira said.
While the transition to electronic checklists began with the Final Optical Systems group (now TAO), Oliveira said a recent major user of NEO has been the Target Diagnostic Systems group. “They really cultivated it for the last year and a half,” she said. “Almost every single checklist that they use on a day-to-day basis is in NEO.”
“NEO is a very good idea,” said Target Diagnostic Operator Brandi Lechleiter. “We usually use it to set up the diagnostics for a shot. Our version of NEO is for quick-use procedures for an experienced operator; if more detail is needed it’s kept on a paper form and can be looked up.”
Lechleiter praised NEO’s customizability and its ability to track the time used for each step in a process. “We can log on to the Internet from the iPad,” she said, “and the ability to take photos on the iPad is very very nice.
“It’s nice not having a huge pile of paper at the end of the day that you have to recycle, and it’s nice to be able to look back and see what procedures were performed in the past, and who did them, and how long they took. NEO allows us to look up what and when, and that makes metrics a lot easier to do.”
Target area operator Mike Sullivan uses an iPad with NEO software to support Steve Keesee as he exchanges diagnostic “snouts” in a diagnostic instrument manipulator. Credit: James Pryatel As NEO’s usage grows, additional enhancements are in the works, such as automatically logging into a “seating chart” that lists the location and installation date of specific components. Also under development is a “dynamic checklist” which will enable a manager to combine multiple checklists from different groups to create a single job. This will allow one workflow through the entire job, with each group receiving a trigger for its portion of the work, minimizing errors and coordination delays.
“NEO still has some capabilities that we would like to see implemented,” Lechleiter said, “but with more and more use we’re starting to like it even more.”
