The Inference Era: What HumanX 2026 Revealed About AI’s Next Billion-Dollar Bottleneck
Training got all the headlines. Inference is what will define the next decade of product design.
I left Moscone Center on Thursday thinking about plumbing. Not literally — but about the invisible infrastructure that makes the things we design actually work, and what happens when that infrastructure becomes the product.
HumanX 2026 was the kind of conference where the most important conversation wasn’t the one on the main stage — it was the one happening in the hallway right after it ended. This year, that conversation was almost always about the same thing: inference. Not models. Not benchmarks. Not which foundation lab is winning the capability race. The question everyone kept circling back to was far more uncomfortable: how do you actually run this at scale without bleeding money?
For those of us who spend our days thinking in user flows and component libraries, the topic might feel distant. It isn’t. The economics of AI inference are quietly reshaping every product decision we make — from which features get built to how quickly users hear back from an AI assistant to whether a startup survives its Series A.
A Paradox That Should Alarm Every Product Person
Here’s the headline nobody put on a slide but everyone was living with: AI token costs have dropped by roughly 280x over the past two years, and yet the average enterprise AI budget has ballooned from $1.2 million annually in 2024 to $7 million in 2026. Some Fortune 500 companies are reporting monthly AI inference bills in the tens of millions of dollars.
The paradox is almost too clean: intelligence is getting cheaper. Deploying intelligence is getting more expensive. And the companies that don’t understand the difference between those two things are going to spend the next two years very confused about their burn rate.
“Don’t confuse the deflation of commodity tokens with the democratization of frontier reasoning.”
That line came from Gartner’s research published just weeks before the conference — and it echoed through basically every “AI in production” conversation I had at HumanX. The cheap tokens are for simple, high-frequency tasks. The expensive compute is for the agentic, reasoning-heavy workflows that teams actually want to build. And as agentic AI scales — where models don’t just answer questions but execute tasks end-to-end across live systems — the cost profile changes entirely.
Jensen’s Three Waves, and Why We’re at the Expensive One
One of the clearest frameworks to emerge from the conference was Jensen Huang’s articulation of three waves of AI, which GTMnow captured well in their post-conference recap. Wave one was extraction: “What is this? Where is that?” Wave two was reasoning: “summarize this, analyze that.” Wave three — where we are now — is autonomous execution: “build this for me, figure it out and report back.”
Each wave is exponentially more computationally demanding than the last. An extraction query might cost fractions of a cent. An agentic workflow that reads documentation, integrates with three APIs, iterates on a result, and then writes back to a system of record? That can consume hundreds of times more tokens — and that compute runs constantly, not just when a user clicks a button.
Deloitte puts inference at roughly two-thirds of all AI compute in 2026, up from one-third in 2023. And the market for inference-optimized chips is projected to hit $50 billion this year alone. We have moved from a world where the interesting infrastructure question was “can you train a model?” to one where it’s “can you serve a model efficiently at scale, to millions of users, 24 hours a day, without going broke?”
What This Looks Like in Practice
There were some fascinating real examples of this playing out. NVIDIA’s partnership with Together AI and Decagon — serving production inference for a multimodel voice stack on Blackwell GPUs — reportedly cut cost per query by 6x compared to proprietary closed-source models. The combination of speculative decoding, smart caching, and auto-scaling pushed response times under 400 milliseconds even under heavy load. That’s not a research result. That’s a deployed product.
Meanwhile, Anthropic (which apparently hit a $3 billion revenue run rate, per hallway conversations) is betting that owning silicon is the key to controlling cost-per-token at scale — partnering with Google and Broadcom to build custom AI chips. At their growth trajectory, that math starts to matter enormously.
The broader structural shift is this: we’re moving toward hybrid AI infrastructure — centralized data centers for heavy reasoning workloads combined with localized on-premise systems for lower-latency inference and redundancy. Open models are narrowing the gap with proprietary ones fast enough that running capable AI locally is now a realistic option for enterprises. A year ago that wasn’t really true.
Why Designers Should Actually Care About This
I’ll be honest: when I first started hearing inference economics discussed this intensely at a conference, I felt a familiar instinct to tune out. Infrastructure is for engineers. Tokens are for engineers. My job is the experience, the interface, the thing the user touches.
But spending a few days at HumanX, surrounded by product leaders and founders who are actually shipping AI products at scale, made me see this differently. The economics of inference are now a design constraint. And design constraints are exactly where designers do our best work.
Think about it this way. When you’re designing a feature that involves an AI-powered background agent, something that continuously monitors, synthesizes, and surfaces insights, you’re not just designing a UX pattern. You’re making decisions that have direct cost implications. An always-on monitoring agent consumes compute continuously, even when no user is actively in the product. The design choice of “this runs in the background” is also, invisibly, a line item on the infrastructure budget.
This is already changing how some product teams work. At the session “AI Without Design Is Just Faster Dysfunction” (which had the best title of the conference, honestly), the conversation kept returning to the same tension: teams are shipping AI features that their users love and that their finance team cannot explain. The experience and the economics are getting designed in separate rooms, and it shows.
For designers and product people: three things to act on
Learn the difference between token-cheap and token-expensive interactions. High-frequency, simple tasks (search, lookup, basic summarization) can and should route to smaller, cheaper models. Reserve frontier reasoning for the moments that genuinely require it. This is model orchestration, and it’s increasingly a product design decision, not just a backend one.
Design with cost as a feedback signal. If a feature’s inference costs scale faster than its user value, that’s a design problem, not just an engineering one. The most elegant AI features are often the ones that accomplish the most with the fewest tokens. Constraint is good design.
Understand what “always-on” really means. Agentic features that run continuously represent a fundamentally different cost model than on-demand features. Before shipping something that lives in the background, ask: What’s the hourly cost of this feature when nobody is actively using it? The answer might reshape what you build.
The Horizon
There’s something I keep coming back to from the week. At a conference called HumanX, you’d expect the conversations to be mostly about what AI can do for people. And they were. But the most honest, eyes-open conversations were about what it costs to actually build those things, and who’s going to pay for it.
Gartner forecasts that inference costs for frontier models will drop 90% by 2030. But in the same breath, they note that token consumption will rise faster than costs fall, meaning total inference spend will keep climbing. The commodity intelligence is becoming nearly free. The intelligence that actually matters, multi-step reasoning, agentic workflows, and spatial understanding, remains scarce and expensive.
What this means for designers is still unfolding. But I left Moscone with a working hypothesis: the most important design skill of the next few years might not be prototyping, or systems thinking, or even AI fluency in the abstract. It might be the ability to design intelligence frugally, to make AI products that feel effortless to the user while being genuinely efficient under the hood.
We’ve been designing for screens for decades. We’re about to start designing for compute budgets, too. And honestly? I think that’s going to make us better designers.
What’s your read on AI inference as a product constraint? I’d love to know what your team is navigating — leave a comment or find me on X.
References
Gartner (March 2026). By 2030, inference on a 1T-parameter LLM will cost 90% less than in 2025. gartner.com
GTMnow Newsletter (April 10, 2026). AI Conference HumanX Takeaways and Frameworks. thegtmnewsletter.substack.com
Deloitte (February 2026). Why AI’s next phase will likely demand more computational power, not less. deloitte.com
Oplexa (March 2026). AI Inference Cost Crisis 2026: Why Your AI Bill Is Exploding. oplexa.com
NVIDIA Blog (February 2026). Leading Inference Providers Cut AI Costs by up to 10x With Open Source Models on Blackwell. blogs.nvidia.com
Digitimes Asia (March 2026). AI shifts to inference as costs rise, memory constraints emerge. digitimes.com
VAST Data (December 2025). 2026: The Year of AI Inference. vastdata.com
HumanX 2026 (April 2026). Official conference agenda and tracks. humanx.co
LLM Stats (April 2026). AI Trends 2026 — LLM Statistics & Industry Insights. llm-stats.com
Brilliant synthesis. The paradox you've identified — 280x token cost drop yet 6x budget increase — is the clearest signal that centralized inference doesn't scale economically. From a distributed AI architecture standpoint, your three-wave framework maps perfectly to where compute needs to live: Wave 1 (extraction) belongs entirely at the edge. Wave 2 (reasoning) is hybrid. Wave 3 (agentic execution) requires intelligent orchestration across edge and cloud. The teams that win won't just design for compute budgets — they'll design inference topology as a first-class product decision. Route cheap tokens locally, reserve cloud for frontier reasoning, and let the architecture itself become the cost optimization layer.