Just now, AI guru Andrej Karpathy expressed his strong liking for the DeepSeek OCR paper, stating:
<blockquote>
I quite like the new DeepSeek-OCR paper. It's a great OCR model (maybe slightly worse than dots), yes, data collection and so on, but that doesn't matter anyway. The more interesting part for me (especially as someone who is core to computer vision, temporarily masquerading as natural language) is whether pixels are better suited as input for LLMs than text. Are text tokens wasteful and bad as input?
</blockquote>
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/6b8f7c78-2804-4731-ae05-dd914f807cbb.png?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/6b8f7c78-2804-4731-ae05-dd914f807cbb.png"/>
Still unsure about the situation, check out my article from yesterday: DeepSeek's bombshell: 10 times compression rate, 97% decoding accuracy! Contextual optical compression makes its debut.
Karpathy believes that aside from the model itself, the DeepSeek paper raises a more thought-provoking question: Is pixel input superior to text input for LLMs? Are text tokens both wasteful and poor?
He further speculates that perhaps all LLM inputs should only be images. Even pure text content should be rendered as images before being input to the model.
Karpathy provided four core reasons supporting this idea:
1. Higher information compression efficiency
Rendering text as images can achieve higher information compression, which means shorter context windows and higher operational efficiency.
2. More universal information flow
Pixels are a far more universal information flow than text. They can represent pure text, easily capture bold and colored text, and even any charts and photos.
3. Default implementation of powerful bidirectional attention
Pixelated input can naturally and easily default to using bidirectional attention for processing, which is more powerful than autoregressive attention.
4. Complete elimination of Tokenizer
Karpathy does not hide his disdain for Tokenizers. He believes Tokenizers are an ugly, independent, non-end-to-end stage. They introduce all the ugliness of Unicode and byte encoding, inherit a lot of historical baggage, and bring security and jailbreak risks (e.g., continuous byte issues).
He gave an example where Tokenizers can cause two characters that look completely identical to the human eye to be represented as two completely different tokens internally. A smiley face emoji, in the model's view, is just a strange token rather than a real smiley face made up of pixels, which prevents the model from leveraging the transfer learning advantages brought by its visual information. Tokenizers must disappear, he emphasized
Karpathy summarized that OCR is just one of many vision-to-text tasks. Traditional text-to-text tasks can be completely restructured into vision-to-text tasks, but the reverse is not true.
The future interaction model he envisions may be: the user's input (Message) is an image, while the output of the decoder (Assistant's response) is still text. It is currently unclear how to realistically output pixels or whether it is necessary to do so.
<h2>Core Controversy: Bidirectional Attention and Image Patching</h2>
Regarding Karpathy's viewpoint, AI scholar Yoav Goldberg raised two questions:
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/1c40786e-0e45-4511-b81b-dd1280e29555.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/1c40786e-0e45-4511-b81b-dd1280e29555.jpeg"/>
<ol>
<li>
Why is it said that images can easily obtain bidirectional attention, while text cannot?
</li>
<li>
Although there is no Tokenization, isn't splitting the input image into patches a similar and possibly uglier processing method?
</li>
</ol>
Karpathy provided an explanation.
He responded that, in principle, nothing prevents text from using bidirectional attention. However, for efficiency, text is usually trained in an autoregressive manner. He envisions that a fine-tuning phase could be added in the middle of training to handle conditional information (such as the user's input message, since these Tokens do not need to be generated by the model). But he is unsure if anyone has done this in practice. Theoretically, to predict the next Token, even the entire context window could be bidirectionally encoded, but this would prevent parallelization of training.
Finally, he added that perhaps this aspect (bidirectional attention) is not strictly a fundamental difference between pixels and Tokens, but rather that pixels are usually encoded, while Tokens are typically decoded (borrowing terminology from the original Transformer paper).
<h2>Musk: 99% of the Future is Photons</h2>
At the end of this discussion, Elon Musk also appeared in the comments section and provided a more futuristic judgment:
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/db4bf304-8ae7-4f6c-ad0c-c2a4b773422a.jpeg?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/db4bf304-8ae7-4f6c-ad0c-c2a4b773422a.jpeg"/>
In the long run, over 99% of the input and output of AI models will be photons. Nothing else can be scaled.
Musk's comment was not made lightly. He further supplemented it with a hardcore cosmological explanation to illustrate why he believes &#34;photons&#34; are the ultimate scaling solution.
In simple terms, the vast majority of particles in the universe are photons
The main source of these photons is the Cosmic Microwave Background (CMB). According to calculations, the photon density of the CMB is about 410 photons per cubic centimeter. Multiplying this density by the vast volume of the observable universe (with a radius of about 46.5 billion light-years) yields an astonishing number of photons contributed solely by the CMB: approximately 1.5 x 10⁸⁹.
In contrast, the number of photons emitted by all stars (starlight) and other sources (such as neutrino background, black hole radiation, etc.) is negligible.
The underlying physical fact revealed here is that photons have an unparalleled advantage in terms of magnitude. This may be the fundamental logic behind Musk's belief that the future input and output of AI will be dominated by photons.
AI Cambrian, original title: &#34;DeepSeek OCR Paper Goes Viral! Andrej Karpathy: I really like it; Musk: 99% of the future will be photons.&#34;
Risk Warning and Disclaimer
The market has risks, and investment requires caution. This article does not constitute personal investment advice and does not take into account the specific investment goals, financial situation, or needs of individual users. Users should consider whether any opinions, views, or conclusions in this article align with their specific circumstances. Investment based on this is at their own risk

DPSK

The DeepSeek OCR paper has sparked heated discussions, with AI expert Andrej Karpathy expressing his strong appreciation for it, considering it an excellent OCR model. He explored the potential of pixels as input for LLMs, arguing that pixels may be superior to text, presenting four main reasons to support this view, including higher information compression efficiency, versatility, advantages of bidirectional attention processing, and criticism of Tokenizers

DeepSeek OCR paper ignites the internet! Andrej Karpathy: I really like it; Musk: 99% of the future is photons