Tencent has published technical information about ARGUS, the software it uses to collect telemetry and diagnose errors on large GPU clusters. Tencent describes ARGUS as “a low-overhead, fine-grained, always-on tracing and real-time analysis system for large-scale training workloads.”
Purpose and architecture
ARGUS is intended to gather data and help debug problems that arise during AI training. According to Tencent, the system is organized into three software layers:
- a Python layer for scheduling and data preparation;
- a framework layer for phase orchestration;
- a GPU runtime layer for kernel execution tracing.
The design emphasizes continuous, fine-grained tracing with low runtime overhead, enabling real-time analysis of training behaviour.
Deployment and case studies
Tencent reports deploying ARGUS on a production cluster of over 10,000 GPUs for more than six months. The paper demonstrates the system’s practical effectiveness through five real-world case studies, which include:
- diagnosing compute stragglers (slow GPUs),
- detecting degradation in communication links,
- analyzing amplification of pipeline bubbles,
- uncovering JIT compilation blocking,
- identifying compute stragglers that were masked by communication symptoms.
Tencent also cites several large training runs where ARGUS was used, including a 4,096-GPU video-language model training job (likely a “HunyuanVideo” model), a 512-GPU audio-model training job, and a 12,960-GPU Mixture-of-Experts (MoE) training job (likely related to a Hunyuan large language model).
Significance
Systems like ARGUS are indicative of complex, large-scale infrastructures where in-house tooling for telemetry and diagnostics is justified. While ARGUS itself resembles tools one would expect at any major frontier AI developer, its deployment and sustained operation on a 10,000+ GPU production cluster for over six months point to the maturity of Tencent’s training environment. Tencent states that ARGUS ran stably alongside production training and played a key role in rapid fail-slow detection and performance optimization.
Further reading
Tencent’s technical write-up is accompanied by an arXiv paper titled: ARGUS: Production-Scale Tracing and Performance Diagnosis for over 10,000-GPU Clusters.
