AWS offers AI-in-a-box for enterprise datacenters

re:invent Many businesses and government agencies require that all sensitive data stay on-premises for legal or security reasons. If those orgs want to work with AI, they can't rely on regular public clouds, but now they can let AWS build and manage AI hardware and software in their datacenters. Announced Tuesday at the company's re:Invent conference, AWS AI Factories is a fully managed solution where enterprises provide the datacenter and power while the house of Bezos installs its hardware and software under their roof. They operate like a private AWS Region, using customers' existing datacenter space, power, and network links while AWS brings in and manages its own racks of infrastructure, including compute, storage, database, and AI services, all running locally. Customers will not need to worry about acquiring hardware, installing it, or buying or building software platforms for their AI models. All they need is the physical space in a datacenter and enough power capacity to juice all those GPUs. That will save orgs a lot of time and expertise. Customers can use AWS tools like the Amazon Bedrock foundation model builder or SageMaker machine-learning platform, as well as some high-end hardware, such as the company's Trainium3 AI accelerators and Nvidia GPUs like the current-gen B200 and GB200, or next-gen GB300 and B300. It will use a petabit-scale, non-blocking network to connect the GPUs and offer Amazon FSx for Lustre and Amazon S3 Express One Zone storage technology. AWS does not yet support NVLink Fusion, which is a high-speed, chip-to-chip interconnect, but says support will arrive in its future Trainium4 chips. Amazon didn't disclose how much this will cost, so we don't yet know whether it will be more expensive than installing one's own hardware and software from scratch. "The AI factories operate exclusively for each customer and it helps them with that separation, maintaining the security and reliability you get from AWS while also meeting stringent compliance and sovereignty requirements," AWS CEO Matt Garman said in his keynote. Garman said the work was inspired by the service's efforts to spin up private, secure AI capabilities for the Kingdom of Saudi Arabia’s AI endeavors, working with the ironically named AI platform company Humain. The company is helping build an "AI Zone" in the country, which will have up to 150,000 AI chips and dedicated AWS infrastructure. According to Garman, the AWS partnership provides Saudi Arabia with high performing infrastructure, models, and AI services like SageMaker and Bedrock, while meeting the kingdom’s security, privacy, and responsible AI standards. “This type of work has sparked some interest in others: large government organizations in the public sector who are interested in a similar concept,” Garman said. “We sat back and asked ourselves. Could we deliver this type of AI zone to a broader set of customers. Maybe even something that could leverage customers' existing datacenters?” Deploying to Amazon's cloud is a pain in the AWS younger devs won't tolerate AWS and Google build a fix for multi-cloud barriers they said didn't exist AWS joins Microsoft, Google in the security AI agent race AWS: How do you do, fellow kids? Please watch our keynotes in Fortnite In building its own AI factories, AWS faces some stiff competition. Dell’s AI Factory with Nvidia was introduced in early 2024, and its promises of an edge to datacenter solution have captured billions in sales. In May, the company boasted 3,000 customers for its AI Factories, and last week, Dell said it had shipped $15.6 billion in AI servers year to date [PDF]. HPE’s own private AI cloud product, which is also backed by Nvidia, won adoption by more than 300 new logos during the quarter ended July 31, it announced during its last earnings call [PDF]. Lenovo has also announced [PDF] a massive upswing in the sale of its infrastructure solutions, up 24 percent year over year in the quarter ended September 30, including "high double digit growth" of AI servers. Amazon's timing may be a bit of a mismatch, as polling from analysts at Forrester and Gartner indicates that the purse strings are tightening on AI spending as customers want a solid track to a return on their investment with these systems, which can cost millions of dollars to stand up, not to mention the price to power, maintain, and operate them. Naveen Chhabra, principal analyst at Forrester, speculated that AWS has likely been building this for several quarters, and suggested AWS is playing into the challenges that customers face when deploying AI infrastructure, including cooling, long lead times for products, the piecemeal approach to architecture, and data sovereignty. According to a November 21 report that Chhabra co-wrote, the revenue achieved as a result of AI spending is lagging and customers are taking notice. “With free cash flow tightening, interest rates still high, and even OpenAI’s CEO warning of a dot-com-style bubble, the sector faces a reckoning,” Forrester stated. The analyst group also posited that the looming chip shortage is causing limited and uncertain access to memory across devices, while pushing vendors to rethink their architecture. Chhabra sees many customers moving AI workloads to the cloud to combat these challenges, but that won't work if you are required to keep all of your data on-prem, which is the target market for AWS AI Factories. ®