--- title: "The limits of \"squeezing\" silicon: How to keep GPUs from being \"idle\"?" type: "News" locale: "en" url: "https://longbridge.com/en/news/294582344.md" description: "The AI industry has sparked a \"chip-making war,\" with giants like Google and OpenAI laying out plans for self-developed chips. At the same time, improving the utilization rate of already deployed GPUs has become a new focus, giving rise to a trillion-level AI infrastructure market. Valuations of inference platforms such as Baseten, Fireworks, and vLLM have soared, attracting attention from giants and venture capital, marking a shift in the focus of computing power from training to efficient inference" datetime: "2026-08-01T11:38:18.000Z" locales: - [zh-CN](https://longbridge.com/zh-CN/news/294582344.md) - [en](https://longbridge.com/en/news/294582344.md) - [zh-HK](https://longbridge.com/zh-HK/news/294582344.md) --- # The limits of "squeezing" silicon: How to keep GPUs from being "idle"? The AI industry has suddenly entered a "chip-making war." On July 20, The Information reported that Google is developing a server chip codenamed "Frozen v2," planning to directly embed part of the Gemini model's architecture into the silicon. A month earlier, on June 24, OpenAI unveiled its first inference chip Jalapeño, co-developed with Broadcom, which took only nine months from design to tape-out. In addition, Anthropic, Zhipu AI, and DeepSeek have also been reported to be making related arrangements. However, not every company has the funds or the capability to recreate a chip. Therefore, a market is emerging that is attracting the attention of capital and giants: AI infrastructure. The underlying logic is that there is still room for optimization in the already deployed data centers and GPUs. The AI inference platform Baseten has seen its annual revenue grow 20 times, with its valuation skyrocketing from $2.1 billion to $13 billion. In the same sector, Fireworks has quadrupled its valuation to $17.5 billion within seven months, with annual revenue exceeding $1 billion. Recently, this sector has also welcomed a strong force. The open-source inference engines "Gemini" vLLM and SGLang announced their commercialization in succession, with seed round financing exceeding $100 million, gathering almost all the names of giants and top venture capitalists in the AI industry, marking the beginning of a strong competition. Thus, "GPU utilization" has become the latest keyword in Silicon Valley, leading to a new trillion-dollar market. This article is a video rewrite; you are welcome to watch the following video. The focus of large models is shifting from training to inference, which will pose completely different requirements for computing power. Training is extremely costly, but once the training is completed, the task comes to a halt. In contrast, inference is different; whether it is a chatbot or an agent, the more AI applications there are, the more requests the GPU needs to handle, and these requests are continuous, leading to an explosive growth in inference demand > Zhu Banghua > > Co-founder of RadixArk > > As new applications gradually increase, the total amount of inference will be much larger than before. As everyone continues to expand applications and the intelligent boundaries of models, the demand for inference will grow increasingly. It has now reached a point where it is very difficult to find cards on the market, and every company is short on cards. The current situation is that every company on the market is short on cards, and for wealthy giants, the most direct method is, of course, to continue building data centers and purchasing more GPUs. Tech giants like Meta, Google, and Microsoft continue to increase capital expenditures, with a total expected to exceed $1 trillion this year. Jensen Huang predicts that by 2030, global annual investment in AI infrastructure will reach $4 trillion. But Wall Street is concerned. > Ethan Xu > > Former Microsoft Energy Strategy Manager, Director of Breakthrough Energy Research > > Wall Street's concerns are not unfounded; we have past experiences to learn from. The burst of the internet bubble, for example, saw a large amount of capital invested in fiber optic construction and internet companies. Although they realized their value in the long term, significant problems arose in the short term. So, if we cannot significantly increase computing power in a short period, what other methods can satisfy the rapid development needs of AI? The answer is: improve GPU utilization. The underlying industry behind this is called AI Infra (Artificial Intelligence Infrastructure Layer). First, let's clarify a misconception about GPUs from the perspective of the overall architecture of AI Infra: you might think GPUs are very busy, but in fact, most of the time, GPUs are quite "idle." The GPU is just one part of the entire AI system. From the moment a user request enters the system, it needs to go through network transmission, resource scheduling, model loading, memory management, and finally complete inference and return results. Throughout this process, if any link encounters a bottleneck, the GPU can only stop and wait. A study released by Carnegie Mellon University (CMU) in April this year conducted fine-grained telemetry on 756 GPUs in a large academic AI cluster for 31 days, covering six models including A6000, L40S, A100, H100, and the latest B200. The results found that many times the GPU was in a state called "execution-idle." How serious is this "seemingly busy yet actually idle" situation? In the cluster they observed, nearly 20% of execution time and about 11% of energy consumption were wasted waiting. More concerning is the inference scenario, where Azure Code workloads consumed up to 65% of their energy in this idling state, and OpenAI's Chat-type requests reached 52%. So, why are GPUs not fully utilized? We first need to understand the four-layer architecture of AI infrastructure, as each layer can affect GPU utilization. The first layer: Power Infrastructure, which refers to electricity. This layer determines whether the GPU can operate continuously and stably. The second layer: Hardware Infrastructure, which refers to the cards. In addition to GPUs, this now includes high-bandwidth memory (HBM), high-speed interconnects (NVLink, InfiniBand), and CPUs. This layer determines the theoretical computational limits that can be provided. The third layer: System Software, including CUDA, compilers, communication libraries, memory management, and low-level kernel libraries, determines whether each computation can be executed to the physical limits of the hardware. The fourth layer: Service Orchestration, which is responsible for coordinating GPU resources, deciding which tasks to prioritize, and dynamically scheduling between different services. Inference engines, training frameworks, request scheduling, and resource management all belong to this layer, with representative projects like vLLM and SGLang, which are also the areas of most innovation in recent years After clarifying the architecture of AI Infra, let's return to our question: why are so many GPUs underutilized, and what are the main bottlenecks? Theoretically, any issue at any of the four levels can affect GPU utilization: insufficient power and cooling will force the GPU to run at lower frequencies; if hardware interconnects can't keep up, the GPU will spend a lot of time waiting for data instead of processing it; inadequate system software optimization will prevent each computation from reaching the physical limits of the hardware; and improper request scheduling will cause computations that could be merged to be split apart, and reusable results to be recalculated, leading to waste. However, the nature of the problems at these four levels is different. The energy infrastructure design layer and the computing hardware layer are "hard" problems, where the cycles for power, cooling, and chip modifications are measured in years, and the optimization potential is rapidly reaching its peak. The system software layer and service orchestration layer are "soft" problems, essentially engineering and algorithmic issues, where optimizations can lead to several times the performance improvement. In this context, the industry's attention is quickly shifting towards these two "soft" issues. > Chen Zhenlin (Richard) > > RadixArk Member of Technical Staff > > Although costs primarily occur at the physical layer, most of the work needs to be focused on the software layer and service orchestration. Because investing human resources in this area can bring the greatest optimization potential. To give a specific example. A NVIDIA GB200 NVL72 cabinet has a procurement cost of about 4 million dollars. However, if the software stack does not properly schedule and optimize, the actual utilization of the GPU may only be 50%, which means only 2 million is being used, while the remaining 2 million dollars are wasted in the form of electricity costs, depreciation, and opportunity costs. Conversely, if utilization can be increased from 50% to over 90%, the effect is equivalent to having an additional cabinet for free. > Chen Zhenlin (Richard) > > RadixArk Member of Technical Staff > > The best performance at the software layer will lead to better services and better token consumption. By rewriting and optimizing this layer, it helps the reasoning engine and training framework to interact better, including the scheduling of post-training processes like reinforcement learning, which is particularly popular now, to improve GPU utilization. This is why, when hardware costs reach this level, every optimization at the software layer becomes a direct business issue. Not long ago, netizens discovered that the largest proportion of employees at Anthropic are now engineers related to AI Infra, which is closely related to this logic. > Chen Zhenlin (Richard) > > RadixArk Member of Technical Staff > > Anthropic's reasoning team now has about 200 people. Over the past three years, they have gone from extreme instability in infrastructure to a state of extreme stability, while also strengthening cost management and optimization, which directly impacts their revenue, allowing them to achieve profitability in the second quarter. OpenAI, Anthropic, and Google place such AI Infra teams internally because it has become one of the competitive barriers for major cutting-edge laboratories. However, for smaller model teams or startups that lack the funding for optimization, two open-source frameworks have become their pillars: SGLang and vLLM. > Chen Zhenlin (Richard) > > RadixArk Member of Technical Staff > > This is why open-source communities like SGLang emerged two years ago. You need such new frameworks to redefine this wave of new hardware system combinations to better adapt to them. Understanding the four-layer architecture and the two "soft" issues that need optimization, what should be done specifically? Currently, all optimization work can be summarized as answers to four questions: Which calculations do not need to be redone? Which waits can be eliminated? Which computing power is not fully utilized? Which resources are not coordinated? The SGLang and vLLM mentioned earlier revolve around these questions, but their design approaches are slightly different. vLLM has gained widespread attention for proposing PagedAttention, emphasizing conventional and general inference deployment scenarios. SGLang, on the other hand, starts from the inference execution process, placing greater emphasis on computational reuse and system-level optimization. ▍Extreme Reuse of K/V Cache There is a "very silly" fact in large model inference: when the same piece of text is repeatedly fed to the model, it has to compute from scratch each time. This is a significant flaw in practical business, especially in agent workflows, where the same tool description may be called dozens of times, starting from zero each time. The industry now refers to this as "inference tax," which is somewhat similar to an IQ tax, as it essentially represents wasted money. Can this tax be avoided? Of course, it can. Anthropic previously implemented an optimization that directly reduced costs by 90%; something that originally cost ten dollars now only costs one dollar after optimization. The core idea behind this is called KV Cache reuse, which simply means storing what has been computed once and using it directly next time without recalculating from scratch. During decoding in the Transformer, each layer of self-attention maps historical tokens into Key/Value tensors, referred to as KV, and the cache is the storage. With KV Cache, during the prefill phase of the first request (referring to the process from user inputting the prompt to generating the first token), the system can store each token and each layer's K/V from the prompt. When decoding each new token afterward, only the KV of the new token is computed and appended to the cache, while the K/V of historical tokens is reused directly However, KV Cache has a drawback. While it can reduce the computational load of decoding and lower the first token latency (TTFT), it comes at the cost of occupying GPU memory. When the context is long and there are many user requests, KV Cache can consume a large amount of memory. Once the memory is insufficient, it becomes difficult for the model to continue improving concurrency and speed, so it often becomes the main bottleneck of the inference service. Its core idea is to use a data structure called "Radix Tree" to organize the massive requests of KV Cache. You can think of it as a family tree with shared prefixes, where requests with the same beginning share the same branch, and only when the content starts to diverge do new branches grow. > Zhu Banghua > > Co-founder of RadixArk > > In today's world of intelligent agents, there are often many shared system prompts, followed by different user prompts. Under the same user prompt, everyone continues to ask questions, so the tree structure keeps extending downwards. This is also the visionary aspect of the SGLang paper, which anticipated this pattern as early as 2023. As more users engage in intelligent agent programming, the benefits brought by the Radix Tree, including the prefix cache system of SGLang itself, become increasingly significant. For example, Cache-aware scheduling is about processing similar requests together. You can think of the operation of a large model as a restaurant kitchen. If the kitchen is currently processing three consecutive orders of spicy hot pot, many seasonings and preparations can be reused, making it very efficient. Even if there are differences, they are just small variations like mild, medium, and spicy. But if the order changes to spicy hot pot, sushi, steak, spicy hot pot, sushi, pizza, spicy hot pot, the chef has to switch back and forth, which reduces the opportunities for reuse. Therefore, SGLang uses Cache-aware scheduling to group similar prefix requests together, rearranging the order to 3 spicy hot pots, 2 sushi, steak, and pizza, making it easier for the cache to hit and further improving efficiency. There is also a mechanism called Eviction, which discards the least likely caches to be used when memory is insufficient, similar to managing your phone's memory. Photos and documents from ten years ago may not be as valuable as data from the last few months So commonly used system prompts, high-frequency RAG documents, and recently used dialogue prefixes will be retained, while when the memory is insufficient, the KV Cache that has not been used for a long time, has low reuse value, and is unlikely to be accessed again will be eliminated. Another technology that makes AI Infra appear very intelligent is called distributed cache-aware load balancer. Large-scale services usually have many GPUs. If a long prompt's KV Cache is on GPU 1, but the next request with the same prefix is sent to GPU 3, then GPU 3 does not have this cache and has to recalculate. Therefore, SGLang will try to send requests to the GPU that "already has notes," which can significantly reduce the amount of repeated calculations. ▍Waiting: Keep the GPU Busy In the current market, memory prices can rise by 90% in a quarter, high-end memory is out of stock until 2027, and even with money, you might not be able to grab a GPU. However, half the time, these expensive cards are just "waiting." The first type of waiting is between requests. The earliest inference engines operated on a "queue system," where requests came one by one, and one was calculated before the next. Later, it evolved into "batch processing," where enough requests were gathered to be processed together, but batching also requires waiting, which does not solve the waste problem. Moreover, if there are both short and long requests in a batch, the short ones have to wait for the long ones to finish before they can be processed, leading to a poor user experience for short requests. The current mainstream approach is called Continuous Batching, which no longer waits for a full batch to be ready before processing. Instead, it operates like a bus that can pick up and drop off passengers at any time, allowing new requests to board and completed requests to disembark at any time, keeping the GPU fully loaded. This transformation can increase the actual throughput of the GPU by 2 to 4 times. In addition to waiting for requests, there is also waiting during the computation phase. The inference of large models actually consists of two stages: "reading in" and "generating." The reading stage (Prefill) requires quickly processing a large amount of text and demands high computational power. The generating stage (Decode) involves outputting one character at a time, which requires less computation, but each character written must traverse the entire memory, making it extremely sensitive to memory bandwidth In the past, these two parts were mixed together in the same GPU for execution, resulting in Prefill waiting while Decode was computing, and Decode waiting while Prefill was computing, dragging each other down. The current approach is called Prefill/Decode separation, which splits these two stages onto different GPUs, each using the most suitable hardware configuration, and passing intermediate results through high-speed networks. For example, DeepSeek adopts this solution, where 32 GPUs form a minimum computing unit during Prefill "reading the question," and when it actually starts "answering word by word," the task is handed over to another batch of GPUs, allowing both groups to work independently without interference. Next, let's talk about the third major issue: the problem of maximizing computing power. ▍Computing Power: "Release Twice as Much" When NVIDIA's H100 was released, the Tensor Core computing power improved several times compared to the A100, and the entire industry expected AI inference to grow synchronously and achieve performance leaps, but the reality was not so. The main reason behind this is that GPUs spent a lot of time on data transfer and were limited by serial decoding, preventing computing power from being fully utilized. There are two main directions for solving this issue. One is low-precision computing. When large models are running, the GPU memory mainly contains three things: model weights, intermediate activation values, and KV Cache. The precision of these three parts can be set independently. Using smaller data formats to store or compute any of these parts can reduce memory usage and leverage faster low-precision computing power on the GPU. In suitable scenarios, it can even release twice as much computing power space. The other is speculative sampling (Speculative Decoding), which means using a small model (draft model) to "guess" the next few words, and then having the large model perform a one-time verification. If the guess is correct, it generates multiple words in one computation. To make an analogy, the large model is a very capable but busy teacher, while the small model is a fast assistant. Previously, every time a student wrote a word, they had to ask the teacher if that word was correct, and only after the teacher confirmed could they write the next word. But with speculative sampling, the assistant first helps the student write a small draft, and the teacher reviews this small section at once. If everything is correct, it is all approved; if there is an error in the middle, they start over from the point of error. In this way, the generation speed can be increased by 2 to 3 times in the best case. The inference engine plays a crucial role here, integrating these new capabilities into the system at the first opportunity, bridging software and hardware, and optimizing collaboration. > Zhu Banghua > Co-founder of RadixArk > Zhu Banghua > Co-founder of RadixArk Finally, let's talk about an important optimization of SGLang in reinforcement learning: Miles. ▍Resource Collaboration: Adapting Miles After RL Training Reinforcement learning has become the core training method for the new generation of models, including OpenAI o1 and DeepSeek R1, which has also led to a significant change in the form of AI training. Traditional model training is a "single-physical" task: feed data, update parameters, feed data again, and update parameters again, with the GPU performing the same type of work throughout the process. In this case, the hardware configuration and scheduling strategy only need to be optimized for the task of "training." However, reinforcement learning requires the integration of inference, evaluation, and parameter updates—three completely different computations running in a loop, each with vastly different hardware requirements. > Zhu Banghua > Co-founder of RadixArk > To be precise, reinforcement learning must perform online inference. Its load is that I first let the model generate a lot of content to be answered, then score based on that content, update the model, answer some new questions, and then score again. So it is inherently a process of alternating training and inference. If traditional training frameworks are used to run reinforcement learning, they will compete for resources within the same batch of GPUs, with training waiting during generation and generation waiting during training, leading to a significant waste of computing power during phase switching and mutual waiting. > RadixArk Member of Technical Staff > If we purely ignore training and only do inference, we may not be able to bet on the future. Because ultimately, many people may still want to train their own models and then perform inference. Therefore, if we want to build an overall AI software infrastructure, we believe that both sides need to be addressed Compared to traditional training frameworks, the biggest difference of Miles is that it considers "training" and "inference" within the same system. This is very important in reinforcement learning, as we mentioned earlier, where inference and training are an alternating process. If not coordinated well, a lot of computing power can be wasted. > Chen Zhenlin (Richard) > > RadixArk Member of Technical Staff We have discussed so much technical theory because this layer is really important for the development of AI large models. It aims to extract the limits of "silicon" and systematically release the computational efficiency of the entire AI system. As we mentioned earlier, SGLang and vLLM are both open-source architectures, mainly relying on a group of researchers and engineers who maintain them in their spare time and contribute out of passion. However, when the whole world faces a shortage of computing power and needs to "squeeze" the limits of silicon to release more computing power, relying solely on the collaboration of the open-source community is no longer sufficient to meet the huge demand. Therefore, the founding teams of SGLang and vLLM have chosen to formally start businesses to push the technology forward more quickly and with greater commitment. Recently, Jensen Huang led the establishment of an open-source AI alliance, which includes model companies, chip companies, and AI infrastructure projects like SGLang. > Zhu Banghua > > Co-founder of RadixArk Therefore, when AI infrastructure capabilities transition from the exclusive resources of a few leading model giants to a public good that any startup can directly access, the threshold for engaging in AI work will be significantly lowered. This, perhaps, is the most important significance of "extracting the limits of silicon." We also hope to see that after the full release of infrastructure dividends, more and more teams with new ideas, new architectures, and new application scenarios will no longer be slowed down by infrastructure, bringing AGI closer to us. ### Related Stocks - [SOXL.US](https://longbridge.com/en/quote/SOXL.US.md) ## Related News & Research - [Google Piles On Off-Balance-Sheet Risk to Fuel Its AI Chip Push](https://longbridge.com/en/news/294586329.md) - [LIVE MARKETS-AI's not that fun and easy. Just look at the SOX](https://longbridge.com/en/news/294098193.md) - [How the 'Nostradamus of AI' failed to spot a market crash](https://longbridge.com/en/news/294580418.md) - [Meet the Gen Z AI whiz at the center of a hedge fund meltdown](https://longbridge.com/en/news/294425537.md) - [You can now call out LinkedIn AI slop using a new button](https://longbridge.com/en/news/294395160.md)