Is Google about to have a major breakthrough?
Recently, Sebastian Borgeaud, the head of pre-training for Google DeepMind's Gemini, made a significant revelation in an interview—
In the coming year, the field of large model pre-training will see major technological innovations in two key areas: &#34;long context processing efficiency&#34; and &#34;context length expansion.&#34;
At the same time, the three giants of Google Gemini—Jeff Dean, Oriol Vinyals, and Noam Shazeer—rarely appeared together on stage, and their discussions showed an astonishing consistency with Sebastian's content.
Many forward-thinking ideas, shining with the light of wisdom, provoke deep thought.
No wonder Google remains that giant.
<h2>Google Executive Excitedly Predicts: Core Secrets of Large Models Have Been Unlocked</h2>
Sebastian Borgeaud, the head of pre-training for Google DeepMind's Gemini, stated in a recent interview that significant innovations in pre-training technology aimed at improving long context processing efficiency and further expanding model context length are expected in the coming year.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/bcb554be-6197-495c-ae6e-3baf1127252a.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/bcb554be-6197-495c-ae6e-3baf1127252a.jpeg"/>
He also revealed that they have made some very interesting discoveries in attention mechanisms recently, which may reshape their research direction in the coming months.
He expressed great excitement about this.
Moreover, he made a striking statement: Scaling Law has not perished; it is merely evolving!
Sebastian Borgeaud is the head of pre-training for Gemini 3.
This is his first blog interview, providing us with an in-depth understanding of the laboratory thinking behind Gemini 3—where the changes are happening and why the current work is no longer about &#34;training models,&#34; but about building a complete system.
<h2>Behind Gemini 3: The Future of AI is Accelerating Towards Us</h2>
After an unexpected leap, a system was born.
&#34;If I am honest with myself, I think... we have come further than I thought we could.&#34;
Sitting in front of the microphone, Sebastian Borgeaud's tone was calm, but this statement was like a stone thrown into a lake, creating infinite ripples.
Why has Gemini 3 achieved such a significant performance leap?
Sebastian's answer seems simple: &#34;better pre-training and better post-training.&#34;
However, behind this understatement lies a fundamental cognitive shift.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/d9f2102f-c21d-40c7-a536-8c3a3df0ec1a.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/d9f2102f-c21d-40c7-a536-8c3a3df0ec1a.jpeg"/>
&#34;We are no longer just building a model,&#34; he said slowly, &#34;I believe that at this point, what we are really building is a system.&#34;
And this is the key to the disruptive progress of Gemini 3.
People often imagine that from one version of Gemini to the next, there are always some groundbreaking &#34;secret weapons.&#34; But the truth revealed by Sebastian is that progress comes from the aggregation of countless subtle improvements. It comes from the &#34;knobs&#34; and optimizations discovered day after day by a large team.
He also pointed out the potential shift in the paradigm of AI development: in the past, it seemed we had unlimited data and could expand datasets at will; now, we are moving towards a &#34;data-limited&#34; system.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/77c3ada3-dffc-4ec3-a203-c14d96533da5.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/77c3ada3-dffc-4ec3-a203-c14d96533da5.jpeg"/>
This means that future AI must learn to utilize limited data resources more efficiently to build more complex systems engineering.
This is the core value of model architecture research.
This shift will force the entire industry from a rough model of &#34;great effort yields miracles&#34; to a refined model of &#34;meticulous craftsmanship.&#34;
The future competitive focus will no longer be &#34;who has the largest data center,&#34; but rather &#34;who has the most efficient learning algorithms,&#34; &#34;who has the most elegant model architecture,&#34; and &#34;who can extract more wisdom from limited data.&#34;
The brain architecture of Gemini 3—the Mixture of Experts (MoE) model—represents a clear path for the development of LLMs: shifting from the pursuit of mere &#34;size&#34; to the pursuit of &#34;large and efficient, large and intelligent.&#34;
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/5323a671-6c01-4a82-ac1e-164115d2c505.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/5323a671-6c01-4a82-ac1e-164115d2c505.jpeg"/>
<h2>The Next Stop for AI—Long Contexts, Efficient Retrieval, and the &#34;Cost Revolution&#34;</h2>
Looking ahead to the technological frontier of the next few years, Sebastian pointed out exciting directions that will collectively shape the capabilities and forms of the next generation of AI.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/e5d1f55b-0774-43d2-be11-60d894f7f574.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/e5d1f55b-0774-43d2-be11-60d894f7f574.jpeg"/>
1. Long Contexts: From &#34;Short Memory&#34; to &#34;Massive Workbench&#34;
The ultra-long context capability brought by Gemini 1.5 is already a revolution. Sebastian predicts that innovations in this area will continue to accelerate. &#34;I believe that in the next year or so, we will see more innovations in making long contexts more efficient and further extending context length.&#34;
This is not just about &#34;remembering more.&#34; Ultra-long contexts will turn the model into a true digital workbench: it can simultaneously load an entire codebase, multiple research papers, and long conversation histories, and conduct coherent analysis, reasoning, and creation within them
This provides unprecedented possibilities for complex multi-step tasks, in-depth research, and creation, serving as a key infrastructure for advancing more powerful &#34;intelligent agents.&#34;
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/c558669b-5b7e-44f2-9e9f-ff273304596f.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/c558669b-5b7e-44f2-9e9f-ff273304596f.jpeg"/>
2. The Evolution of Attention Mechanisms
Sebastian specifically mentioned that there have been &#34;some truly interesting discoveries&#34; regarding attention mechanisms recently, which he believes will shape a significant amount of research in the coming months. He is &#34;personally very excited&#34; about this.
This suggests that the attention mechanism, considered the cornerstone of current large models, still has enormous room for improvement.
More efficient, powerful, or feature-rich attention mechanisms could significantly enhance the model's understanding, reasoning, and computational efficiency from the ground up, serving as an important lever for pushing performance boundaries.
3. The Return of Retrieval: Teaching Models to &#34;Look Up Information&#34;
Sebastian previously led the &#34;Retro&#34; project, which explored enabling models to retrieve external knowledge bases during training and inference, rather than memorizing all knowledge in parameters. He believes this direction is far from outdated.
&#34;I deeply believe that the long-term answer is to learn this capability in a differentiable manner.&#34; This means that future models may combine retrieval and reasoning more naturally, dynamically acquiring information from vast knowledge sources for thinking, rather than relying on later &#34;patched&#34; search tools.
This could make models more precise and timely, potentially breaking through the knowledge capacity limits imposed by parameter scale.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/59ea7c50-d3b7-4acd-8431-c61837f98bae.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/59ea7c50-d3b7-4acd-8431-c61837f98bae.jpeg"/>
4. The &#34;Revolution&#34; of Efficiency and Cost
An increasingly prominent challenge is that, with the surge in users, the deployment and service costs of models have become crucial. Future research will no longer solely pursue peak performance but must also focus on how to make powerful models &#34;cheap and easy to use.&#34;
<h2>I Still Don't See an End</h2>
As the interview neared its conclusion, when we shifted the focus back to this researcher standing at the forefront of the AI wave, Sebastian's response revealed a calm optimism and pure enthusiasm.
Despite discussing real issues such as data bottlenecks and cost challenges, Sebastian remains confident about the overall progress of AI.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/ad0b75de-d9ee-41a1-bd4c-79752a382452.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/ad0b75de-d9ee-41a1-bd4c-79752a382452.jpeg"/>
&#34;I have mentioned this many times, but there are indeed so many different things that will compound together, and there are many directions with room for improvement. I really don't see any end to this kind of work stopping us from making progress.&#34;
This confidence is not blind. It stems from what he has witnessed as a frontline navigator: a wealth of subtle yet certain areas for improvement, as well as the untapped creative vitality of the entire field
In his view, at least in the coming years, this momentum of progress will not slow down.
<h2>Scale is No Longer a Myth: Noam Shazeer's First Cold Water After Returning</h2>
Next, there was a discussion among three big names: Jeff Dean, Noam Shazeer, and Oriol Vinyals.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/1a7478fb-3542-4835-8602-123569954217.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/1a7478fb-3542-4835-8602-123569954217.jpeg"/>
During the meeting, we could clearly feel that Noam is no longer the radical who floors the accelerator.
He hardly initiates discussions about &#34;disruption&#34; and rarely uses those grandiose terms that excite people. In fact, he unusually talked multiple times about the pace of research and development, system stability, and how to operate in the long term.
As the pioneer of the Transformer, he once led the development of large models from scratch; now, as large models are advancing rapidly, he has stopped to warn: the answers provided by large models come too quickly, and the frequency of self-checking is too low.
He has repeatedly emphasized in public:
Current models are not lacking in &#34;intelligence,&#34; but in the ability to think continuously and to repeatedly correct themselves in complex tasks.
In other words, the scale of large models is still important, but it is no longer the only variable that determines everything.
When Noam discusses reasoning, he clearly shifts the focus from &#34;can it be stronger&#34; to &#34;can it be more stable.&#34;
This is the first time he has shown this judgment in public since returning to Google.
It does not sound like a denial of the past, but more like an acknowledgment of a fact: the path that purely relies on stacking parameters to keep moving forward is nearing its limits.
<h2>Consensus Among Top Scientists: A High Score Cannot Define the Future of AI</h2>
Noam mentioned a term: Slow Thinking.
It is not just about simply slowing down the pace of research and development, but about repeatedly questioning whether it is worth it, whether it is expensive, and whether it can be scaled and replicated.
At this point, intelligence is no longer an abstract capability, but an engineering expense that requires long-term investment, needing to be discussed alongside CPU and hard drives in the accounting books.
Once this step is taken, many past standards for measuring AI, such as Benchmark, are slowly becoming ineffective.
Throughout the meeting, the three top scientists did not show excitement about rankings, nor did they define progress with &#34;who won.&#34;
Rankings are good at measuring instantaneous performance but struggle to answer long-term questions like &#34;can it run continuously.&#34;
What Noam and Jeff repeatedly emphasized is precisely the latter: whether the model is reliable, whether it has transferability, and whether it can continuously self-correct in complex tasks.
These capabilities are difficult to compress into a neat number.
<h2>Gemini is Treated as a System, Not a Model</h2>
In this discussion, the term &#34;System&#34; appeared frequently, which is clearly not a rhetorical device
Noam and Jeff deliberately avoided the term &#34;a stronger model&#34; when describing Gemini, instead repeatedly emphasizing that it is a &#34;system&#34; that can operate long-term and iterate continuously.
&#34;System&#34; and &#34;model&#34; may sound similar, but the underlying logic is vastly different.
A &#34;model&#34; is more like a one-time achievement, a momentary performance in the laboratory; whereas a &#34;system&#34; resembles infrastructure, focusing on stability, scalability, and the ability to quickly fix errors and continue functioning.
For someone like Noam, who places extreme emphasis on architectural design, development pace, and engineering constraints, the ability of a system to operate robustly for ten or twenty years is naturally more important than the speed of a single response.
In their view, Gemini is not a trophy for the victor, but a form of intelligence that is &#34;long-term usable.&#34;
Because of this, the entire conversation did not rush into product promotion or engage in a tense &#34;benchmarking against a model.&#34;
It was more about emphasizing externally: Google is not pursuing a fleeting finished product, but rather a reusable and continuously evolving intelligent industrial system.
Risk Warning and Disclaimer
The market has risks, and investment requires caution. This article does not constitute personal investment advice and does not take into account the specific investment goals, financial situation, or needs of individual users. Users should consider whether any opinions, views, or conclusions in this article align with their specific circumstances. Investing based on this is at one's own risk

GOOGL

GOOG

GGLS

GGLL

Sebastian Borgeaud, the head of pre-training for Google's DeepMind's Gemini, revealed that significant technological innovations in long context processing efficiency and context length expansion will occur within the next year. He also mentioned interesting discoveries in attention mechanisms that could reshape research directions. Borgeaud emphasized that the Scaling Law has not disappeared but is evolving

- Google DeepMind's Sebastian Borgeaud announced significant advancements in pre-training technology for large models, focusing on long context processing efficiency and context length expansion.  
- The discussion among Google's top researchers highlighted the importance of evolving attention mechanisms and integrating retrieval capabilities to enhance model performance, emphasizing a shift from merely scaling up to building more efficient systems.  
- These innovations are expected to redefine AI's future, prioritizing stability, reliability, and the capacity to iteratively learn from complex tasks over sheer parameter size.

Scaling Law is not dead! Core executives of Gemini reveal that Google already has a disruptive key