ThunderAgent Boosts Synthetic Data Generation with 2.5x Speedup

Together.ai has unveiled ThunderAgent, a system designed to optimize agentic inference for synthetic data generation. By rethinking how inference workflows are scheduled, ThunderAgent delivers a 2.5x throughput improvement on single nodes and scales near-linearly across multi-node GPU clusters. The system, which has been accepted as a Spotlight paper at ICML 2026, promises to streamline large-scale synthetic data pipelines critical for modern AI applications.

ThunderAgent’s key innovation lies in treating each agentic workflow as a schedulable program rather than a series of independent requests. This approach eliminates inefficiencies such as KV cache thrashing—where memory is wasted repeatedly storing and evicting conversation histories during pauses for tool calls. The result is a dramatic reduction in latency and a significant improvement in resource utilization.

Addressing Bottlenecks in Agentic Inference

Agentic inference, where AI agents perform multi-step reasoning and tool use during runtime, plays a pivotal role in generating synthetic datasets. Unlike static prompts, agentic systems simulate dynamic, multi-turn scenarios, often interacting with external tools or environments. However, existing inference engines like SGLang and TensorRT-LLM falter at high concurrency due to KV cache inefficiencies. These engines treat each model call as independent, leading to cache evictions and costly recomputations when agents resume workflows.

ThunderAgent solves this by adding a program-aware scheduling layer that tracks each workflow’s execution phase, memory footprint, and cluster placement. By pausing low-priority workflows during memory pressure and intelligently redistributing them across GPU nodes, ThunderAgent minimizes cache thrashing while balancing workloads. Tests on an 8-node H100 cluster showed a 2.4x speedup over SGLang Gateway, with ThunderAgent achieving 2,248 steps per minute as cluster size scaled to 64 GPUs.

Implications for AI and Synthetic Data

Efficient synthetic data generation has become increasingly critical for AI research and deployment. Natural datasets often lack the complexity needed for training agentic systems, necessitating large-scale synthetic alternatives. ThunderAgent powers pipelines like Together.ai’s CoderForge, where hundreds of concurrent agents simulate multi-turn coding scenarios to generate high-quality data. This aligns with recent advancements, such as Apple’s environment-free synthetic data generation for API-calling agents and ontology-guided frameworks for rare-event data augmentation.

ThunderAgent’s open-source design also makes it accessible for broader adoption. It integrates seamlessly with existing inference backends using OpenAI-compatible APIs and works alongside optimizations like quantization and speculative decoding. Together.ai emphasizes that the only client-side change required is adding a simple program ID field.

Market and Research Impact

As synthetic data generation becomes a cornerstone of AI development, systems like ThunderAgent could significantly reduce costs and improve efficiency. Recent studies suggest that agentic systems can generate datasets for single-digit dollar costs per run, making them highly economical for domains like safety-critical perception, web automation, and scientific research. ThunderAgent further amplifies these benefits by improving throughput and scalability without requiring additional hardware investment.

ThunderAgent is poised to shape the next generation of agentic inference systems. With its ICML 2026 recognition and applicability to real-world pipelines, it will likely attract adoption from both academic researchers and industry practitioners. For those running large-scale agentic workloads, the system offers a “free lunch” speedup using existing hardware—a compelling value proposition in today’s compute-intensive AI landscape.

Developers and researchers can explore ThunderAgent via its GitHub repository or access the detailed research paper.

Image source: Shutterstock