In the era of AIGC, a new computing paradigm is being explored and developed. Accelerating the AI inference process with CPUs has proven to be the optimal path, validated through practice. CPUs excel at executing inference tasks involving extensive sequential computations. According to a QYResearch report, the global market for AI inference servers is estimated to be around 74 billion yuan in 2023, with projections to grow to 267.6 billion yuan by 2030, representing a compound annual growth rate of 18.9%.

The continued growth in demand for high-performance computing resources in AI inference will further stimulate CPU innovation.

What Kind of CPU Does AI Inference Require?

The server head is the core component of a server, primarily responsible for managing and coordinating server nodes in a computing cluster. In high-end AI servers, a common practice is to equip 2 CPUs for every 8 GPUs, allowing the CPUs to coordinate and manage the associated acceleration hardware, ensuring efficient collaboration and accelerating data processing and AI inference.

The CPU is the most critical component within the server head, responsible for executing instructions and processing data. Its performance and capabilities directly determine the overall performance and processing efficiency of the server. Therefore, evaluating the overall CPU performance is crucial when selecting an AI inference server. Choosing a CPU with optimal architecture and specifications that perfectly match application requirements, such as performance, core count, and thread count, is an art, especially when considering different application scenarios and workloads.

AI inference involves using trained models to predict and analyze new data, placing stringent demands on servers for high performance, scalability, low latency, and security.

1. Higher Frequency, Higher Performance

CPU frequency is a vital performance indicator in AI inference, as it directly impacts computation speed and efficiency. A CPU with a high clock speed offers faster computation, essential for handling complex AI inference tasks. Based on practical experience, selecting a CPU with a high clock speed ensures seamless performance and efficiency when processing vast amounts of data and complex computations. Additionally, multicore processors excel at multitasking, significantly enhancing AI inference speed.

AMD's fourth-generation EPYC processor platform (Genoa), built on the Zen 4 microarchitecture, targets scenarios such as AI, multi-cloud workloads, high-performance computing, and real-time analytics. The AMD Genoa CPU boasts a base clock speed of 3.51GHz. In GeekBench 5 benchmarks, the Genoa CPU scores 1460 for single-core and 96,535 for multi-core, demonstrating a notable 28% improvement in multi-core performance over its predecessor, the EPYC Milan 7763.

The AMD Genoa CPU's higher core frequency and ability to support multiple cores simultaneously at peak frequency make it an ideal choice for AI inference.

2. Large Cache, Multiple Benefits

CPUs equipped with a Level 3 cache typically excel at handling massive data volumes and high concurrency tasks, significantly enhancing multitasking and multithreading performance. For AI inference, which demands high-performance computing, a Level 3 cache is preferable. During AI inference, processing decisions or recognitions from trained models involves complex logic, control flow tasks, and extensive data. The combination of fast L1 and L2 caches with a large L3 cache results in higher cache hit rates (indicating that most data can be found in the cache without accessing main memory), thereby accelerating AI model processing and analysis.

Compared to the previous-generation EPYC Milan, AMD Genoa doubles the L2 cache size to 1MB per core and supports a shared 32MB L3 cache across eight cores. The AMD Genoa-X system further enhances this by offering a shared 96MB L3 cache across eight cores, bolstering AI inference capabilities.

3. Large and Fast Memory for Stable and Swift Inference

For AI inference applications, CPU memory speed is crucial, as it directly affects model loading, data processing, and result output speeds. Higher memory speeds enable faster data processing and enhanced AI inference efficiency. Additionally, memory capacity cannot be overlooked. As models continue to expand, memory capacity must keep pace. CPUs with more memory channels support larger capacities and bandwidths, better accommodating DDR5 memory, vital for boosting overall processor performance. Synchronized growth in memory capacity and speed is essential for handling AI inference tasks effectively.

All AMD Genoa series CPUs support DDR5 memory speeds of up to 4800MT/s, theoretically supporting up to 920Gbps of memory bandwidth in dual-socket configurations. Each AMD Genoa CPU features 12 memory channels supporting up to 24 DIMM modules. These attributes ensure ample memory bandwidth and capacity for AI inference. AMD Genoa's memory design balances efficiency and stability, leveraging high-speed DDR5 memory and an optimized memory architecture to ensure stable system operation and rapid data processing under heavy loads. In summary, AMD Genoa CPUs excel in memory performance.

AMD, uniquely offering solutions across GPU, CPU, and FPGA platforms, stands out with its comprehensiveness and ability to design and optimize for specific application needs, positioning it at the forefront in the AIGC era. AMD EPYC CPUs are particularly favored by enterprises for AI inference. The launch of AMD's fourth-generation Genoa EPYC 9004 series elevates AI inference capabilities to new heights.

AMD EPYC: A Versatile Performer

AI inference technology finds diverse applications, ranging from financial security and weather forecasting to medical diagnosis and gaming entertainment. Selecting the most suitable server and CPU model for specific applications and scenarios is crucial for optimal performance.

When choosing a CPU for AI inference, comprehensive considerations should include computation speed, latency, AI optimization capabilities, cost-effectiveness, and software ecosystem to ensure optimal performance and efficiency. AMD EPYC CPUs, including the 9334, 9454, and 9534 models, each demonstrate exceptional capabilities tailored for AI inference applications.

These AMD EPYC CPUs share similarities, including high clock speeds, multiple cores, and substantial bandwidth, offering remarkable cost-effectiveness and energy efficiency. For instance, the Zen 4 architecture enhances AMD EPYC 9004's instructions per clock cycle by approximately 14% compared to its predecessor, coupled with higher clock speeds, significantly boosting performance. Furthermore, the EPYC 9004 series boasts increased core and thread counts (up to 50% more than its predecessor, with a maximum of 96 cores) and supports simultaneous multithreading, offering both high concurrency and low latency. The introduction of DDR5 memory, 12 memory channels, and up to 128 PCIe 5.0 lanes in the EPYC 9004 series creates a high-speed data transmission backbone, enhancing efficiency and speed for massive data processing and high-performance computing tasks. Overall, AMD EPYC 9004's impressive performance accelerates AI inference.

Each AMD EPYC CPU model (9334, 9454, 9534) excels in its unique areas, catering to diverse user needs and specific workloads. The AMD EPYC 9334, a 2.70GHz 32-core processor, combines versatility with high performance, efficiency, robust virtualization capabilities, and exceptional thermal management, ideal for data-intensive processing and standard enterprise infrastructure applications. The AMD EPYC 9454, a 2.75GHz 48-core processor supporting DDR5-4800 memory and equipped with 256MB cache, operates at 290W, making it perfect for handling vast data volumes and complex computations. The AMD EPYC 9534 stands out for its balance, featuring a 2.45GHz base clock speed, 64 cores, and 256MB L3 cache. It offers superior performance and efficient system resource allocation, making it the most cost-effective choice for AI inference.

AI's Endless Journey: CPU Upgrades Continue

The future of AI inference holds much promise, encompassing reasoning analysis, creative generation, emotional intelligence, and multimodal technologies, ultimately aligning with human intelligence. This underscores the crucial role of servers and CPUs in supporting AI inference.

As data volumes soar and algorithmic complexity escalates, challenges in cost and technical complexity intensify, necessitating the integration of additional cores, GPUs, and other components. While enhancing processing efficiency, energy consumption and related issues must also be addressed. Riding the AI wave, AMD EPYC processors strive for excellence in high-performance computing, security enhancements, energy efficiency optimization, and adaptability to emerging technologies.