December 3, 2020
LLNL, along with partners Intel, Supermicro, and Cornelis Networks, have deployed “Ruby,” a high performance computing (HPC) cluster that will support NIF’s inertial confinement fusion (ICF) experiments, perform functions for the National Nuclear Security Administration (NNSA), and boost the Laboratory’s COVID-19 research.
Funded by NNSA’s Advanced Simulation and Computing (ASC) program, the Laboratory’s Multi-programmatic and Institutional Computing (M&IC) program and the Coronavirus Aid, Relief, and Economic Security (CARES) Act, the 6-petaFLOP peak, Intel Xeon Platinum-based cluster will be used for unclassified programmatic work supporting NNSA’s stockpile stewardship mission, LLNL open science, and the search for therapeutic drugs and designer antibodies against SARS-CoV-2, the virus that causes COVID-19.
“Ruby provides a substantial computing resource in our open collaboration zone, which has experienced a heavy increase in demand due to an uptick in telecommuting and a growth in external collaborations,” said Chris Clouse, acting program director for LLNL’s ASC program. “A resource like Ruby provides a venue for leveraging expertise and tools in the open community for areas that are important to our programmatic missions.”
Early applications for Ruby include large-scale simulations of plasma dynamics and neutron production at LLNL’s MegaJOuLe Neutron Imaging Radiography system and simulations for ICF research conducted at NIF and Sandia National Laboratories’ Z-machine. Additional work will be done on asteroid detection, moon formation, high-fidelity fission, and other basic science discovery through LLNL’s Computing Grand Challenge and Laboratory Directed Research and Development (LDRD) programs.
LLNL COVID-19 researchers also have begun using Ruby to compute the molecular docking calculations needed for discovering small molecules capable of binding to protein sites in the structure of SARS-CoV-2 for drug discovery purposes.
“Ruby is excellent for running the molecular docking calculations,” said LLNL Biochemical and Biophysical Systems Group Leader Felice Lightstone, who heads the COVID-19 small molecule work. “Our early access on Ruby is allowing us to screen about 130 million compounds per day when using the entire machine. As our COVID-19 therapeutic effort moves toward optimizing compounds we have identified as promising, Ruby allows us to maximize the throughput of our new designs.”
Ruby was ranked the 80th most powerful supercomputer in the world when the updated Top500 List was announced on Nov. 16, giving LLNL four of the top 100 on the list. Ruby is a liquid-cooled cluster consisting of more than 1,500 nodes, each outfitted with Intel Xeon Platinum 8276L processors with Intel Deep Learning Boost and 192 gigabytes of memory.
“The Ruby supercomputer is a great example of how the tech industry comes together to solve some of science’s largest challenges,” said Trish Damkroger, vice president and general manager of the high performance computing organization at Intel. “Working closely with LLNL, Supermicro and Cornelis Networks has allowed us to deploy Ruby, which will make significant contributions in the area of COVID-19 research.”
Ruby’s nodes are interconnected using Omni-Path, the purpose-built high-performance network from Cornelis Networks. Spun out from Intel in September, Cornelis Networks continues to partner with Intel to co-optimize HPC cluster solutions based on Xeon and Omni-Path.
“The Cornelis team is thrilled to continue our track record of successful HPC cluster deliveries to NNSA, which date back to 2009,” noted Gunnar Gunnarsson, vice president of Solutions Delivery and Support at Cornelis. “Cornelis is proud to contribute to the continued advancement of open science efforts at LLNL, none more important than the development of therapeutics to eradicate COVID-19.”
San Jose-based IT company Supermicro provided the 26 racks and infrastructure needed, including 376 TwinPro™ Systems and 24 Ultra Systems.
“LLNL chose Supermicro’s HPC servers for the Ruby supercomputing cluster focused on COVID-19 research and advanced computationally intensive workloads for pioneering experimentation and analysis,” said Charles Liang, president and CEO of Supermicro. “Supermicro’s liquid-cooled rack integration with TwinPro and Ultra 2U servers leverage advanced dual top-bin second-generation Intel Xenon Scalable processors for these global research projects. We are committed to helping worldwide efforts to discover leading-edge solutions.”
Follow us on Twitter: @lasers_llnl