For the past 4 years, Mila has been developing Milabench, a transparent graphics processing unit (GPU) evaluation tool enabling data centers to choose the most appropriate hardware for their artificial intelligence research tasks, without being tied to a specific manufacturer.
As a GPU may be more suitable for one use than another, it is crucial for research institutes with their own computing clusters -such as Mila- or for companies to have a neutral, reliable tool at their disposal, enabling them to make informed hardware choices tailored to their own needs.
Existing commercial solutions, often designed by GPU manufacturers, tend to highlight results from optimized calculations in order to promote their own product. However, research centers are rarely able to match these claimed performances as their real-life experiments generally differs from those of the manufacturers’.
Milabench does not seek to promote a specific manufacturer: the experiments taken into account in GPU evaluation are defined in advance without the possibility of modifying them, making it a reliable and brand-agnostic tool.
Informed choices
Milabench was developed in collaboration with Mila researchers using the nearly 1,000 scientific articles affiliated with the institute published in 2023 to analyze the use of GPUs in extremely varied research tasks, depending on the field of expertise.
Indeed, some experiments require a better connection between the various GPUs, others depend more heavily on the transfer rate between the microprocessors (CPU) and the GPU... All these specific cases make it impossible to declare one model “better” than the others in all circumstances.
Thanks to Milabench, research centers can now carry out a reliable and relevant performance-cost analysis and thus achieve maximum performance within their budget.
Optimizing scientific research
Milabench was used by Compute Canada for its latest cluster in service, Narval. It has also been designed to be flexible and fine-tuneable according to the specific needs of computing centers, which vary from one field to another.
Beyond Canada, other computing centers specialized in scientific research -in physics, chemistry, biology, meteorology- can now benefit from this tool for research using artificial intelligence. Milabench is available for free and open source.
Thanks to Milabench, manufacturers will also have greater visibility of researchers' needs, enabling them to make better decisions on what fixes to make (including support for certain popular libraries) to better adapt their products to the needs of cutting-edge AI research.
For more information and results, visit milabenchreport.mila.quebec.
