"how numbers are stored in computers"
In September 2023, a consortium of large technology companies (AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm) introduced a family of standardized low-bitwidth data formats designed to enhance the efficiency of AI training and inference. The MX formats aim to provide a balance between computational performance and model accuracy, particularly in resource-constrained environments.
MX-compliant formats enable Artificial Intelligence (AI) training and inference with lower bit-width arithmetic operations and smaller memory footprints. This drives hardware performance and efficiency gains that can reduce overheads and operational costs. Additionally, standardizing new capabilities and formats that can be implemented across hardware and/or software — while enhancing existing standards such as OFP8 and INT8 — will reduce software and infrastructure costs and any associated costs or overheads with customized solutions.
MX formats utilize a block-based approach, where a group of elements shares a common scaling factor. This method allows for efficient representation of data with reduced precision while maintaining numerical stability.
Each block typically contains 32 elements sharing the same scaling factor.
By reducing the bitwidth of data representations, MX formats decrease memory usage and bandwidth requirements, leading to faster data processing and lower energy consumption. This is particularly beneficial for large-scale AI models and edge computing scenarios.
MX formats are designed to be compatible with existing AI hardware and software stacks. They can be integrated into current systems with minimal modifications, facilitating widespread adoption across different platforms.
The MX specification is released under an open, license-free model through the OCP Foundation, encouraging collaboration and innovation within the AI community. This openness aims to foster the development of new tools and techniques leveraging MX formats.
Emulation libraries such as Microsoft's microxcaling provide tools for experimenting with MX formats in frameworks like PyTorch, enabling researchers and developers to evaluate their impact on model performance.