Artificial Intelligence

LongCat-2.5: Meituan's 1.6T-Parameter Multimodal Model for Long-Horizon Agent Tasks

By 美团技术团队 · Nov 2, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

The techniques for training and serving 1.6T-parameter MoE models, including W4A4 quantization and dynamic expert replication, give concrete patterns for efficient large-model deployment. For developers building long-horizon agents or multimodal systems, the partial rollout strategy and RL stability measures address problems that appear once you scale beyond toy tasks.

Summary

LongCat-2.5 from Meituan's tech team is a 1.6T-parameter MoE model with 48B active parameters per token. It builds on LongCat-2.0 by adding native visual understanding and strengthening agentic capabilities. Visual signals are converted into multi-level discrete tokens through residual vector quantization, allowing the backbone to process images and text in a unified token space. Offline pre-computation of visual token IDs simplifies long-sequence training, and the model shows strong performance on Visual Coding and creative design tasks.

The training pipeline introduces large-scale agentic RL with data synthesized from GitHub and internal code tasks, using fail-to-pass and pass-to-pass validation to label trajectories. To handle long-horizon RL, the team uses partial rollout with logprob clipping to mitigate stale gradients, and applies reward hacking countermeasures like network disconnection and LLM judges. For deployment, W4A4 quantization (weights and activations at 4-bit) yields 43.3% prefill and 42.77% decode throughput gains over W8A8, with TTFT and TPOT down 15.91% and 19.72%. The FSDP configuration achieves 86% higher single-card throughput than DDP in long-sequence SFT, and a training-inference consistency scheme ensures bitwise logprob matching.

Takeaways

— LongCat-2.5 has 1.6 trillion total parameters and activates 48 billion per token.

— Vision input is tokenized via multi-level discrete tokens using residual vector quantization (RVQ) on native-resolution features.

— Agentic RL uses large-scale data synthesis from public GitHub repositories and internal code tasks, with F2P and P2P test validation.

— Partial rollout in RL training saves unfinished trajectories and clips logprob differences from stale checkpoints to stabilize gradients.

— Reward hacking is mitigated by disconnecting network access, clearing git history and caches, and using LLM judges.

— W4A4 quantization of expert weights and activations boosts prefill and decode throughput by 43.3% and 42.77% over W8A8, and cuts TTFT and TPOT by 15.91% and 19.72%.

— FSDP with dual sharding and dynamic expert replication achieves 86% single-card throughput improvement over DDP baseline in long-sequence SFT.

— A training-inference consistency scheme based on IsoExec, GLM-5, and XoRL maintains bitwise logprob matching across training, prefill, and decode.

— The model uses per-cut prefixes instead of per-turn prefixes for online teaching in MOPD, limiting distribution shift in long single-turn responses.

Conclusions

Pre-computing visual tokens offline removes the vision encoder from the training graph entirely, which simplifies parallelization but fixes the tokenizer; the t-SNE evidence from a small MoE does show alignment, yet it's a trade-off against end-to-end visual feature learning.

Partial rollout with logprob clipping tackles stale gradients from mismatched checkpoints; scaling token gradient contribution by historical distance is a practical stability measure that other RL systems could adopt.

Reward hacking countermeasures like disconnecting the network and clearing git history are drastic but necessary; they show that environment sanitation and LLM judges are part of the model's training loop.

Domestic accelerators with under 80GB memory per card force aggressive memory engineering; the 86% FSDP throughput gain likely reflects that constraint, whereas H100-class hardware might not see the same benefit.

Concepts & terms

Mixture of Experts (MoE)

A model architecture where only a subset of parameters (experts) are activated per token, reducing compute cost while keeping a large total parameter count.

Residual Vector Quantization (RVQ)

A method to quantize visual features into discrete tokens by iteratively quantizing residuals, preserving fine-grained detail in the token representation.

W4A4 quantization

Quantizing both weights and activations to 4-bit integers for inference, using native INT4 hardware units; reduces memory and increases throughput compared to W8A8.

PD separation

Splitting prefill and decode into separate serving phases with different parallel strategies, optimizing each phase for its distinct compute and memory characteristics.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗