700 tokens per second: Meta puts Google's custom AI chips into production

Meta Platforms has begun running AI workloads in production on Google’s tensor processing units (TPUs), marking the first time the custom silicon has been used by an external major customer for this specific purpose. The deployment utilizes a new, PyTorch-native version of the SGLang inference engine, a development announced via Google’s developer blog. This move allows Meta to utilize Google's hardware without the need to rewrite code for Google’s proprietary JAX framework, effectively making the TPUs a native option for the industry-standard PyTorch library. A corresponding open-source version of the vLLM inference engine was also released to support this integration.

The shift follows years of friction between major tech firms and custom silicon providers. Google’s PyTorch-on-TPU initiative was originally designed with Meta in mind, as the social media giant resisted migrating its infrastructure to JAX. While Google describes the older PyTorch/XLA bridge as a predecessor to the new TorchTPU, reports indicated that Meta research teams had utilized TPUs through that bridge between 2020 and 2023 before canceling those specific contracts. The current collaboration represents a more mature integration, with Meta’s PyTorch-TPU lead Zongwei Zhou describing the new stack as "remarkably mature" and the result of a "close three-way collaboration" between Meta, Google, and the SGLang team at RadixArk.

Jana van Greunen, director of PyTorch engineering at Meta, emphasized the strategic importance of hardware diversity, stating that the company "cannot be constrained by a single hardware architecture." She noted that the new TorchTPU backend enables Meta to deploy its PyTorch workloads directly to Google’s chips. Although the physical location of these production TPUs has not been disclosed, previous reports suggested negotiations involved Google Cloud TPU rentals starting in 2026, with potential purchases following in 2027, and a subsequent multi-year rental agreement.

Performance benchmarks highlight the competitiveness of the new setup. A tuned vLLM configuration serving Moonshot AI’s 2.8-trillion-parameter Kimi K3 model reportedly achieved over 700 tokens per second for a single user on the seventh-generation Ironwood TPU. This figure, provided by Inferact, a startup founded by vLLM’s creators, was generated using 16 Ironwood chips with speculative decoding. In comparison, the same model on 16 Nvidia GB200s reached 452 tokens per second. Additionally, the new kernels for Kimi K3 and Qwen 3.8 27B deliver approximately 1.4 to 2 times the throughput of the GB200 baseline at small batch sizes when speculative decoding is not applied.

The announcement landed just one day after Amazon revealed it was open-sourcing TorchNeuron, its native PyTorch backend for Trainium2 and Trainium3 chips, at the same PyTorch Conference. Both moves appear designed to reduce the software lock-in associated with Nvidia’s dominant market position. Looking ahead, SGLang-TorchTPU plans to expand model support, starting with DeepSeek V4.1 Flash and Reflection Beam. RadixArk aims to provide Day 0 TPU support for new open models by the first quarter of 2027. Further details on the collaboration are expected to be shared at the SGLang Summit 2026, scheduled for November 12–13 in San Francisco.