New scoop, the protagonist is the Meta flagship model just released yesterday - Llama 4.
<h2>Internal Leak: Performance Below Standards, Pressure to &#34;Optimize&#34; Results?</h2>
The discussion was first ignited by a post from the &#34;Yimuchangfen Di&#34; forum, where the poster claimed to be an internal employee involved in the training of Llama 4 and stated that they had resigned because of it.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/d39a1506-38a3-4465-a039-a300c7502052.png?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/d39a1506-38a3-4465-a039-a300c7502052.png"/>
The post contained a lot of information, mainly mentioning the following points:
<ol>
<li>Performance Bottleneck: Despite the team's repeated efforts in training, the internal model performance of Llama 4 has consistently failed to meet the open-source SOTA (State-of-the-Art) benchmarks, with a noticeable gap.</li>
<li>&#34;Curve Rescue&#34; Strategy: The company leadership proposed mixing various benchmark &#34;test set&#34; data into the training or fine-tuning data in the later stages of training. The purpose of this approach is straightforward - to achieve targets on various metrics and deliver a &#34;beautiful-looking&#34; report card.</li>
<li>Deadline Pressure: This &#34;score brushing&#34; task has a clear timeline - by the end of April. If the targets cannot be met by then, the consequences could be severe.</li>
<li>Poor User Feedback: After the release of Llama 4 (the post was published right after the model was released), many users on X and Reddit have reported that the actual testing results are very poor.</li>
<li>Academic Integrity and Resignation: The poster stated that they have an academic background and cannot accept the practice of &#34;polluting test data to meet standards,&#34; thus they submitted their resignation and explicitly requested that their name not be included in the technical report of Llama 4.</li>
<li>Executive Movements (mentioned in the post): The post also mentioned that Meta's VP of AI resigned for similar reasons. (Note from the blogger: this is a one-sided claim from the post and should be discerned carefully.)</li>
</ol>
This post quickly attracted attention within the community, with discussions on whether this practice violates the basic integrity of AI research and development.
The real situation remains to be observed.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/6779a455-548d-450f-8eea-3b68d6a7e2bf.png?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/6779a455-548d-450f-8eea-3b68d6a7e2bf.png"/>
<h2>External Observation: TechCrunch Questions the &#34;Misleading&#34; Test Version</h2>
Coincidentally, the well-known tech media TechCrunch also published an article, directly pointing out that the performance testing of Meta's new AI model is &#34;a bit misleading.&#34;
TechCrunch's article mainly focuses on Llama 4 (i.e., Maverick) and its performance on the renowned human evaluation leaderboard LM Arena. Maverick did achieve a good second place, but there seems to be more to the story:
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/d130328a-b740-4dc9-9a89-844172db6e9f.png?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/d130328a-b740-4dc9-9a89-844172db6e9f.png"/>
<ol>
<li>
<ol>
<li>Version Differences: The Maverick version submitted by Meta for testing and evaluation at LM Arena may not be the same as the version publicly released for developers.</li>
</ol>
</li>
<li>
<ol>
<li>Official Annotation: Meta actually mentioned this in their announcement and on the Llama official website. They clearly stated that the version used for LM Arena testing is an &#34;experimental chat version,&#34; or labeled as &#34;Llama 4 Maverick optimized specifically for conversational scenarios.&#34;</li>
</ol>
</li>
<li>
<ol>
<li>The Issue of &#34;Optimized for Rankings&#34;: TechCrunch pointed out that although LM Arena itself is not a perfect evaluation tool, AI companies in the past typically did not (at least not publicly) provide a specially optimized version solely to improve ranking. Meta's approach this time is akin to optimizing a version for benchmark testing to boost rankings, while providing developers with an unoptimized &#34;base version.&#34;</li>
</ol>
</li>
<li>
<ol>
<li>Misleading Developers: This operation makes it difficult for developers to accurately estimate the true performance of the model in their actual application scenarios based on ranking. Although benchmark testing has limitations, it should provide a relatively fair reference.</li>
</ol>
</li>
<li>
<ol>
<li>Behavioral Differences: Researchers on the X platform also found that the publicly downloadable Maverick version behaves differently from the version tested on LM Arena. The version on the rankings tends to use more emojis and provides noticeably more verbose answers.</li>
</ol>
</li>
</ol>
<h2>Some Real Tests of Llama 4</h2>
Claiming a recall rate of tens of millions of contexts, the actual performance of the context is far below expectations.
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/a61d4df9-9065-45e1-b126-d12148ad5b14.png?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/a61d4df9-9065-45e1-b126-d12148ad5b14.png"/>
Llama 4 Maverick scored only 16% in the aider multilingual coding benchmark test!
<img src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/ce6e4383-41ab-49f9-8cdf-c6af0f12a31d.png?x-oss-process=image/auto-orient,1/interlace,1/resize,w_1440,h_1440/quality,q_95/format,jpg" alt="" original-src="https://imageproxy.pbkrs.com/https://wpimg-wscn.awtmt.com/ce6e4383-41ab-49f9-8cdf-c6af0f12a31d.png"/>
This article is sourced from: AI Cambrian, original title: &#34;The Big Scoop is Here! Llama 4 is Caught in Ranking Controversy: 'Internal Employee' Posts Accusations, Evaluation Version Allegedly Specially Provided?&#34;
Risk Warning and Disclaimer
The market has risks, and investment requires caution. This article does not constitute personal investment advice and does not take into account individual users' specific investment goals, financial situations, or needs. Users should consider whether any opinions, views, or conclusions in this article align with their specific circumstances
According to this investment, the responsibility is self-borne

Wallstreetcn

Meta Platforms

The big news is here! Llama 4 is embroiled in a ranking controversy: "internal employees" post accusations, and the evaluation version is alleged to be a special supply?