The current global AI industry has transitioned from a competition in model performance to a &#34;life-and-death race&#34; for the large-scale implementation of intelligent agents. &#34;Cost reduction&#34; is no longer an optional optimization but the core lifeline determining whether AI companies can be profitable and whether the industry can break through. Against this backdrop, Inspur Information has launched the Yuan Nao HC1000 ultra-scalable AI server, which has for the first time reduced inference costs to 1 yuan per million tokens. This breakthrough is expected to eliminate the cost barriers of the &#34;last mile&#34; in the industrialization of intelligent agents and will reshape the underlying logic of competition in the AI industry.
Liu Jun, Chief AI Strategist of Inspur Information, emphasized that the current breakthrough of 1 yuan per million tokens is only a temporary victory. In the face of the inevitable trend of exponential growth in token consumption and a tenfold increase in the demand for tokens for complex tasks, the existing cost level still cannot support the widespread implementation of AI. In the future, for AI to truly become a basic resource like &#34;water, electricity, and coal,&#34; token costs must achieve an order-of-magnitude leap from the current level. Cost capability will further upgrade from &#34;core competitiveness&#34; to &#34;survival ticket,&#34; directly determining the life and death of AI companies in the era of intelligent agents.
<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/> https://static001.geekbang.org/infoq/2e/2e4f082af574d2a71f429053bdacf33a.png<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/>
Liu Jun, Chief AI Strategist of Inspur Information
<h2>In the era of intelligent agents, token cost is competitiveness</h2>
Looking back at the history of internet development, the &#34;speeding up and reducing costs&#34; of infrastructure is an important cornerstone of industry prosperity. From dial-up internet billed by Kb, to fiber optic access where 100 Mbps bandwidth became standard, and then to the 4G/5G era where data traffic costs approach zero—each significant reduction in communication costs has driven the explosion of new application ecosystems such as video streaming and mobile payments.
The current AI era is also at a similar critical point. As technological advancements lead to a decline in token prices, companies can apply AI on a large scale to more complex and energy-intensive scenarios, such as from early short Q&amp;A to now supporting ultra-long contexts with multi-step planning and reflection capabilities of intelligent agents... This has also resulted in an exponential increase in the demand for tokens for single tasks. If the rate of decline in token costs does not keep pace with the exponential growth in consumption, companies will face higher expenditure. This indicates that the famous &#34;Jevons Paradox&#34; in economics is perfectly replaying in the token economy.
Data from multiple sources strongly supports the trend of exponential growth in token consumption. The latest data disclosed by Volcano Engine shows that as of December this year, the daily token usage of the Doubao large model under ByteDance has exceeded 50 trillion, an increase of more than 10 times compared to the same period last year, and a growth of 417 times compared to the daily usage when it was launched in May 2024;
Google disclosed in October that its platforms process a monthly token volume of 1.3 trillion, equivalent to an average of 43.3 trillion per day, compared to just 9.7 trillion per month a year ago.
<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/> https://static001.geekbang.org/infoq/63/630b39f9465489f30921f6716ba10c8d.png<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/>
Google announced changes in its token processing volume.
When usage reaches the scale of &#34;trillions of tokens per month,&#34; even a cost reduction of just $1 per million tokens could result in a monthly cost difference of $100 million. Liu Jun believes that &#34;token cost is competitiveness; it directly determines the profitability of intelligent agents. To truly bring AI into a stage of large-scale and inclusive development, token costs must continue to decrease by orders of magnitude from the current level.&#34;
<h2>In-depth Analysis of Token Cost &#34;Black Box&#34;: Mismatched Architecture is the Core Bottleneck</h2>
Currently, the global large model competition has shifted from &#34;blindly stacking computing power&#34; to a new stage of &#34;pursuing the value output per unit computing power.&#34; The value output per unit computing power is influenced by various factors such as energy prices, hardware procurement costs, algorithm optimization, and operational costs. However, it is undeniable that over 80% of current token costs still come from computing power expenditures, and the core contradiction hindering cost reduction lies in the stark differences between inference loads and training loads. Continuing to use old architectures leads to difficulties in optimizing computing power, memory, and network resources simultaneously, resulting in severe &#34;high configuration but low efficiency.&#34;
First, there is a severe inversion of computing power utilization (MFU). The MFU during the training phase can exceed 50%, but during the inference phase, especially for real-time interactive tasks that pursue low latency, due to the autoregressive decoding characteristics of tokens, the hardware must load all model parameters in each round of computation, but only to compute the output of one token, causing expensive GPUs to spend most of their time waiting for data transfer, with actual MFU often only at 5%-10%. This enormous idle computing power is the structural root of high costs.
Second, the &#34;storage wall&#34; bottleneck is amplified in inference scenarios. In large model inference, as the context length increases, the KV Cache grows exponentially. This not only occupies a large amount of memory space but also leads to high power consumption due to memory access intensity. This separation of storage and computation not only incurs data migration power consumption and latency but also requires the use of expensive HBM, which has become an important bottleneck hindering the reduction of token costs.
Third, the costs of network communication and horizontal expansion are becoming increasingly high. When the model scale exceeds the capacity of a single machine, inter-node communication becomes a new bottleneck. The latency of traditional RoCE or InfiniBand networks is much higher than the total bus latency within chips, and communication overhead can account for more than 30% of total inference time, forcing companies to pile on more resources to maintain response speed, thereby increasing total cost of ownership (TCO)
In this regard, Liu Jun pointed out that the core of reducing token costs is not about &#34;making a machine more comprehensive,&#34; but rather about reconstructing the system around the goal: breaking down the reasoning process into finer details, supporting P/D separation, A/F separation, KV parallelism, fine-grained expert splitting, and other computing strategies, allowing different computing modules to be configured concurrently on different cards as needed, fully loading each card, thereby lowering the &#34;cost per card time&#34; and increasing the &#34;output per card time.&#34;
<h2>Based on a brand new ultra-expansion architecture, the Yuan Nao HC1000 achieves a reasoning cost that breaks through 1 yuan per million tokens for the first time.</h2>
Currently, the token costs of mainstream large models remain high. For example, the price for outputting one million tokens for models like Claude and Grok generally ranges from $10 to $15, while domestic large models, although relatively cheaper, are mostly above 10 yuan. With astronomical levels of usage, such high token costs pose a severe ROI challenge for large-scale commercial applications. To break the cost deadlock, a fundamental reconstruction at the computing architecture level is necessary to significantly enhance the output efficiency of unit computing power.
<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/> https://static001.geekbang.org/infoq/19/198914798231d67ba6a74f066b7fb704.png<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/>
Price of one million tokens for mainstream LLMs
(Note: Data from September 26 (the day of the AICC2025 conference), on September 29, DeepSeek announced that the price for V3.2 Exp was reduced to 3 yuan per million tokens.)
To this end, Inspur Information launched the Yuan Nao HC1000 ultra-expansion AI server. This product is based on a newly designed fully symmetric DirectCom ultra-fast architecture, employing a lossless ultra-expansion design that can efficiently aggregate massive domestic AI chips, supporting extremely high reasoning throughput, and achieving a reasoning cost that breaks through 1 yuan per million tokens for the first time, providing an innovative computing power system with exceptional performance to help agents overcome the token cost bottleneck.
<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/> https://static001.geekbang.org/infoq/55/555aec0d257a11bca539feb24d982749.png<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/>
Yuan Nao HC1000 ultra-expansion AI server
Liu Jun stated: &#34;We see that the original AI computing was built with a focus on being large and comprehensive, with all sorts of things included. However, when we focus on the core goal of reducing token costs, we rethink the system architecture design, identify system bottlenecks, and reconstruct a system with a minimalist design
The YuanNerve HC1000 innovatively designs the DirectCom ultra-fast architecture, with each computing module configured with 16 AIPUs, adopting a direct communication design to solve the protocol conversion and bandwidth contention issues of traditional architectures, achieving ultra-low latency; the computing communication is balanced at a 1:1 ratio, enabling global non-blocking communication; the fully symmetric system topology design supports flexible PD separation and AF separation schemes, configuring computing instances on demand to maximize resource utilization.
<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/> https://static001.geekbang.org/infoq/d8/d85af4f80fdabc71116f11fd0119e4be.png<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/>
Fully symmetric DirectCom ultra-fast architecture
At the same time, the YuanNerve HC1000 supports ultra-large-scale lossless expansion, with the DirectCom architecture ensuring a balance between computing and communication. Through deep collaboration of computing networks and global lossless technology, it achieves a 1.75 times improvement in inference performance, and by subdividing the computing process of large models and decoupling model structures, it realizes flexible on-demand allocation of computing loads, with a single card MFU capable of improving performance by up to 5.7 times.
<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/> https://static001.geekbang.org/infoq/a1/a16275931151639c59d7858e36766f3e.png<img src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://www.infoq.cn/news/null"/>
Ultra-large-scale lossless expansion
In addition, the YuanNerve HC1000 provides packet-level dynamic load balancing through adaptive routing and intelligent congestion control algorithms, achieving intelligent scheduling of KV Cache transmission and All to All communication traffic, reducing the impact of KV Cache transmission on Prefill and Decode computing instances by 5-10 times.
Liu Jun emphasized that the current &#34;1 yuan per million tokens&#34; is still far from enough. In the face of the exponential growth of token consumption in the future, to achieve a continuous and order-of-magnitude reduction in the cost per token, a fundamental innovation in computing architecture must be promoted. This also requires product and technology innovation across the entire AI industry, shifting from the current scale-oriented approach to an efficiency-oriented one, fundamentally rethinking and designing AI computing systems, developing AI-specific computing architectures, exploring the development of large model chips, and promoting the innovation of dedicated computing architectures that hardwareize algorithms, achieving deep optimization of software and hardware, which will be the direction of future development

000977

IEIT Liu Jun: The AI industry cannot achieve profitability without reducing costs; a cost of 1 yuan per million tokens is still far from enough!