The relentless demand for artificial intelligence is creating a new bottleneck: GPU underutilization. Across the rapidly expanding landscape of neocloud providers, significant computing power sits idle between training runs and shifting workloads, representing a substantial loss of potential revenue. This challenge has spurred innovation, and FriendliAI is emerging as a key player with a novel solution – InferenceSense – designed to unlock the value of those dormant GPU cycles.
Traditional approaches to utilizing spare capacity often involve spot markets, allowing providers to rent out unused hardware. However, these markets typically offer raw compute power, leaving users to manage the complexities of deploying and optimizing an inference stack. FriendliAI’s approach is fundamentally different: it directly runs AI inference on the idle hardware, optimizes for maximum token throughput, and shares the resulting revenue with the neocloud operator. This paradigm shift promises a more efficient and profitable use of existing infrastructure.
From Academic Research to Industry Innovation
The foundation of FriendliAI and InferenceSense lies in the groundbreaking research of Byung-Gon Chun, a former professor at Seoul National University. Chun’s work culminated in the development of Orca, a technique known as continuous batching. This innovative method dynamically processes inference requests, eliminating the need to wait for a full batch before execution – a process that dramatically improves efficiency. Continuous batching is now considered an industry standard and forms the core of the widely adopted open-source inference engine, vLLM.
Founded in 2021, FriendliAI initially focused on providing dedicated inference endpoints for AI startups and enterprises leveraging open-weight models. The company quickly established a presence on platforms like Hugging Face, alongside major cloud providers such as Azure, AWS, and GCP, supporting over 500,000 open-weight models. InferenceSense represents a natural extension of this expertise, addressing the critical issue of GPU capacity utilization.
How InferenceSense Works: A Seamless Integration
InferenceSense is designed to integrate seamlessly with existing infrastructure. Built to operate on top of Kubernetes – the dominant resource orchestration platform for neoclouds – the system allows operators to allocate a pool of GPUs to a FriendliAI-managed cluster. Operators define the conditions under which GPUs can be reclaimed, ensuring their primary workloads always take precedence. Idle detection is handled directly through Kubernetes itself.
“We have our own orchestrator that runs on the GPUs of these neocloud – or just cloud – vendors,” explains Chun. “We definitely take advantage of Kubernetes, but the software running on top is a really highly optimized inference stack.” When GPUs become available, InferenceSense spins up isolated containers to serve paid inference workloads, supporting popular open-weight models like DeepSeek, Qwen, Kimi, GLM, and MiniMax. The system is designed for rapid response; FriendliAI claims a handoff time of just seconds when a scheduler reclaims a GPU for its primary tasks.
Demand for inference workloads is aggregated through FriendliAI’s direct clients and partnerships with inference aggregators like OpenRouter. The operator provides the capacity, while FriendliAI manages the demand pipeline, model optimization, and serving stack. Operators benefit from a zero-fee, commitment-free model, with a real-time dashboard providing complete transparency into model usage, token processing, and accrued revenue.
Token Throughput: The Key to Maximizing Revenue
The core advantage of InferenceSense lies in its focus on monetizing tokens rather than simply renting capacity. Spot GPU markets, offered by providers like CoreWeave, Lambda Labs, and RunPod, focus on renting out hardware. InferenceSense, however, leverages the existing hardware investment of neocloud operators, allowing them to generate revenue from the actual processing of AI inferences.
FriendliAI asserts that its engine delivers two to three times the throughput of a standard vLLM deployment, although this figure varies depending on the specific workload. This superior performance is achieved through a number of key architectural choices. Unlike most competing inference stacks built on Python-based frameworks, FriendliAI’s engine is written in C++ and utilizes custom GPU kernels, bypassing Nvidia’s cuDNN library. The company has also developed its own model representation layer for efficient partitioning and execution across hardware, along with proprietary implementations of speculative decoding, quantization, and KV-cache management.
This enhanced efficiency translates directly into increased revenue potential for neocloud operators. By processing more tokens per GPU-hour, they can generate a higher return on their unused cycles than they could through traditional capacity rental models.
Implications for AI Engineers and the Future of Inference
For AI engineers evaluating inference solutions, the neocloud versus hyperscaler decision has traditionally revolved around price and availability. InferenceSense introduces a new dimension to this equation. If neoclouds can effectively monetize idle capacity through inference, they will have a stronger economic incentive to offer competitive token pricing.
While it’s still early days, this dynamic could potentially drive down the overall cost of AI inference. As more neoclouds adopt platforms like InferenceSense, the increased competition could lead to downward pressure on API pricing for models like DeepSeek and Qwen over the next 12 months.
“When we have more efficient suppliers, the overall cost will go down,” Chun emphasizes. “With InferenceSense we can contribute to making those models cheaper.”
But what will be the long-term impact on the balance of power between hyperscalers and specialized neoclouds? And how will this new revenue stream influence investment in GPU infrastructure?
Frequently Asked Questions About InferenceSense
A: InferenceSense is a platform developed by FriendliAI that allows neocloud operators to monetize idle GPU capacity by running AI inference workloads. It optimizes for token throughput and shares revenue with the operator, turning unused resources into a new income stream.
A: Unlike spot markets that rent out raw compute capacity, InferenceSense focuses on monetizing the actual processing of AI inferences (tokens). This allows operators to earn more from their existing hardware investment.
A: InferenceSense supports a wide range of open-weight models, including DeepSeek, Qwen, Kimi, GLM, and MiniMax, catering to diverse AI inference needs.
A: InferenceSense is designed to yield immediately when the operator’s scheduler reclaims a GPU. The handoff happens within seconds, ensuring minimal disruption to primary workloads.
A: Continuous batching, pioneered by FriendliAI’s founder Byung-Gon Chun, dynamically processes inference requests without waiting for a full batch, significantly improving efficiency and throughput – a core component of InferenceSense’s performance.
The launch of InferenceSense marks a significant step towards optimizing the utilization of GPU resources and lowering the cost of AI inference. As the demand for AI continues to grow, solutions like this will be crucial for ensuring sustainable and affordable access to the computing power needed to drive innovation.
Share this article with your network to spark a conversation about the future of AI infrastructure!
Keep reading
Discover more from Archyworldys
Subscribe to get the latest posts sent to your email.