Computers and water don’t mix well, anyone who has ever knocked over a glass over a laptop knows. Yet water is advancing in data centers, right down to the computers. Cooling is crucial for data centers, where increasingly compact electronics have more and more computing power, so more and more heat in a small area. Water flowing past cools better than air from a fan. It saves energy and money.
One of the Netherlands’ most powerful computers is located in a data center at the Science Park in Amsterdam: Snellius, the national supercomputer – named after the mathematician Willebrord Snel van Royen. Snellius is a common property, all scientists in the Netherlands can book computing time.
Also read: Supercomputer gives new computing power to science
“Some science can only be done with exceptional computing power. Think of genetics, or climate science, where extensive models are used,” says Valeriu Codreanu, head of high performance computing at SURF, the ICT collaboration organization for education and research in the Netherlands. “For computers: to outcompute is to outcompete: whoever can do the math the best wins. That is why powerful computers are crucial for researchers.”
The computer is ‘super’ because many servers – now about 600 pieces – are linked together that can communicate at lightning speed. Snellius, which was festively inaugurated last year, will be one of the top 100 most powerful computers in the world when it runs at full capacity by the end of 2023. It now occupies almost an entire aisle, is spread over 25 cabinets – ‘racks’ – and has 6.1 petaflops per second of computing power. “It means the computer can do more than six million billion calculations per second,” says Codreanu. There are two expansions planned, it will be 21 petaflop per second.
Deafening sound
It is logical that supercomputers in particular use the most modern cooling technology. “Under our feet lies a system of pipes. It is a closed system, with branches to every rack,” says Walter Lioen, head of research services at SURF, as he opens one of the cabinets. Snellius combines different types of water cooling. “The racks with CPUs, the most commonly used type of processor, are cooled with water in the back door, like a radiator in a car. This is technology that was also used in a previous supercomputer and is now commonplace in data centers.” Fans are still working inside the systems, which together produce a deafening noise.
The rack of GPUs – graphics processors for machine learning are very suitable – has state-of-the-art direct water cooling: copper tubes zigzag through the system, close to the various parts on the printed circuit boards. “This GPU rack is also the very latest for our supplier Lenovo,” says Lioen. “This direct water cooling was not yet available for the racks with CPUs, but it will be available for the new racks that will come at the end of this year.”
Data centers consume energy, only the rack with GPUs from Snellius consumes 70 kilowatts at full capacity – an average laptop requires about 70 watts. Energy consumption has made data centers controversial in recent years. A new data center cannot simply be connected to the energy grid anymore, because there is too little capacity available. They also sometimes consume a lot of sustainably generated energy, which is then not available to make the rest of the Netherlands more sustainable. The arrival of the largest data centers, ‘hyperscales’, is therefore received with skepticism in particular.
The seventh generation
“If you need a lot of computing power, you can’t avoid using a lot of energy,” says Lioen. “Still, the efficiency gains that computers like these are making are impressive. This is the seventh generation scientific supercomputer in the Netherlands. With each generation there is 10 times more computing power, but the energy consumption remains almost the same.”
Energy consumption is strongly related to cooling. For a long time, data centers were only cooled with air. From large air conditioners in the hallways to small fans in the systems themselves. Air is safe, but it boosts energy consumption enormously. Water can store four times more heat per kilo than air. “As a rule of thumb, you can say that on top of the energy to run the system, you need 60 percent extra energy for cooling if you cool with air,” says Lioen. “If you cool with cold water, that’s an extra 40 percent, with warm water like we use here, only 20 percent.”
The risk is not completely zero, something can always break
Walter Lioen Department head
At 22 degrees the water enters Snellius, warmed to about 32 degrees it comes out again. “With cold water cooling you still lose energy on cooling the water, here that happens almost free of charge because the outside temperature is almost always lower than 30 degrees,” says Lioen. “It still needs energy to pump the water around.” Other buildings in the Science Park are heated with the residual heat.
Water sounds like an excellent solution. Still, a leak would be an expensive joke – Snellius costs 20 million euros. “The risk is not completely zero, something can always break,” says Lioen. “But this is really a different story than water cooling that people sometimes tinker with rubber hoses in their computers at home. It goes through copper pipes here and all connections have special no drip-technology.”
Will the water advance further? “There are already systems that are completely submerged, in which case the computers are located in a kind of aquarium with special coolant,” says Lioen. “The equipment is then of course well sealed, but it still seems to me to be a huge hassle for maintenance. I don’t really believe in it yet, but who knows.”