**The Expanding Compute Gap: Why Enterprise AI Spending Is Outpacing Visibility**
Across more than 100 large enterprises, spending on AI infrastructure is accelerating well beyond the ability to track or manage its economics. While most organizations still run their AI workloads on familiar hyperscalers and model-provider APIs, the next dollars are flowing toward specialized compute options that few use today. A striking finding from recent research is that a majority of enterprises intend to switch or add infrastructure providers within the next year, driven by integration needs and total cost of ownership concerns rather than headline token prices. Yet, nearly half cannot rigorously track what their AI compute actually costs, and three-quarters of GPUs sit idle at utilization rates of 50% or less. This widening “compute gap” reveals a critical challenge: enterprises are buying infrastructure faster than they can understand or control its true value.
—
### Key Findings
**1. Ambition Outpaces Production**
Only about one in five enterprises (21%) currently run AI in production at scale. While three-quarters remain in experimentation or limited deployment phases, their spending intentions signal aggressive future build-out. Evaluations over the coming year will focus heavily on AI-specialized clouds—a category nearly unused today—signaling a potential re-platforming wave ahead.
**2. Current Stack Revolves Around Hyperscalers**
Today, most AI workloads run on established public clouds. Google Cloud leads usage at 48%, followed closely by Microsoft Azure and AWS. Major model APIs from Google Gemini, OpenAI, and Anthropic handle nearly all model execution. Specialized AI-cloud providers and on-prem GPU clusters remain marginal.
**3. The Next Dollar Targets Uncharted Infrastructure**
Despite limited use today, 45% of enterprises plan to evaluate AI-specialized clouds within the next year. Interest also runs high in non-NVIDIA accelerators (32%) and next-gen GPUs (28%). This indicates a broad expansion phase, with organizations preparing to move compute off general-purpose clouds.
**4. Provider Switching Is Imminent**
A clear majority (64%) of enterprises plan to switch or add infrastructure providers within 12 months, and 38% expect action within a quarter. While some movement involves new entrants, much of the near-term churn involves reshuffling among major incumbents like Microsoft Azure and Google Cloud.
**5. Buyers Care About Economics, Not Token Price**
When choosing providers, enterprises prioritize integration (41%) and total cost of ownership (35%). Cost per million tokens ranks last (8%), highlighting that decisions are based on operational fit rather than headline pricing. Yet fewer than half can accurately quantify their compute costs.
**6. GPUs Are Largely Underutilized**
83% of enterprises report GPU utilization at or below 50%, with nearly half running below 25%. Only 12% exceed 50% utilization. This inefficiency—coupled with limited measurement—exposes a major gap between capacity owned and value realized.
**7. Measurement Lags Spending**
Fewer than half of enterprises (44%) rigorously track AI compute costs and return on investment. While spending accelerates, visibility into economics remains weak, creating risk as investments scale.
**8. The Next Bottleneck Is Memory Bandwidth**
As inference workloads grow, the shift from GPU compute to memory bandwidth becomes critical. Yet awareness is low, and responses show no clear consensus on solutions. This emerging constraint could shape architecture and costs before enterprises close today’s visibility gaps.
—
### FAQ
**Q: What defines “AI in production at scale” in this report?**
A: The report defines this as organizations where AI is deeply embedded across core operations, rather than limited to experiments or isolated pilots.
**Q: Which providers were included in the “AI-specialized clouds” category?**
A: Examples include CoreWeave, Lambda, Crusoe, and Nebius—providers focused primarily on dedicated AI infrastructure rather than general-purpose cloud services.
**Q: Why is utilization so low across existing GPU deployments?**
A: Contributing factors include over-provisioning, workload mismatches, software inefficiencies, and the experimental nature of many early deployments. The report highlights this as a key symptom of the broader compute gap.
**Q: How certain are these findings given the sample size?**
A: With 107 respondents, the survey offers directional insight rather than precise market measurements. The sample skews toward mid-market and early-stage adopters, so views may differ from hyperscale operator experiences.
**Q: What role does total cost of ownership play in purchasing decisions?**
A: It is the second-most important factor (35%), indicating that enterprises are pursuing economic rationality even when measurement capabilities fall short.
—
### Conclusion
Organizations are caught in a widening compute gap: AI infrastructure investment is surging ahead of the visibility needed to manage it effectively. While hyperscalers remain dominant today, evaluation interest is shifting toward specialized infrastructure—signaling potential re-platforming ahead. Low GPU utilization, weak cost tracking, and unclear economics reveal a market buying faster than it can measure. As the industry shifts from compute constraints to memory-bandwidth challenges, the central lesson is clear. Without better measurement and integration discipline, faster spending risks deepening inefficiency instead of driving value. Enterprises that build visibility before the next wave of infrastructure expansion will be better positioned to convert ambition into sustainable advantage.
