ZICQ
中 Log in / Sign up
Newsroom Compute & Architecture #SageMaker HyperPod #GPU Observability #Distributed Training #CloudWatch #EFA

SageMaker HyperPod Adds Cross-Layer Observability for GPU Training Faults

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Amazon SageMaker HyperPod now provides enhanced observability for GPU training clusters orchestrated by Amazon EKS or Slurm. It combines host, job, accelerator, and EFA telemetry with peer-based GPU outlier detection and static fault rules, allowing teams to trace degradation to a specific job, node, accelerator, and training rank. Default metrics are sampled every 30 seconds, while detailed NVIDIA DCGM, AWS Neuron, and EFA metrics are opt-in to control telemetry volume and cost. AWS reports ap


Amazon SageMaker HyperPod's enhanced observability is now available in every AWS Region where HyperPod is supported. The update does more than add utilization dashboards: it connects host resources, GPU and Neuron health, EFA networking, and job state into a diagnostic chain that can identify the affected job, cluster, node, accelerator, and rank—the global identifier of a distributed-training process.

Capability Set and Boundaries

The enhanced observability stack has three layers:

  • Default metrics: Automatically cover cluster health, CPU, disk, OS utilization, GPU utilization and temperature, and per-job resource consumption. The default collection interval is 30 seconds.
  • Advanced metrics: Optionally add the full Node Exporter suite, detailed NVIDIA DCGM and AWS Neuron accelerator telemetry, and EFA throughput, packet-drop, retransmission, and error metrics. These are most useful for long-running jobs on large clusters, where meaningful patterns emerge over time.
  • Diagnostic summary: Use CloudWatch's DetectGroupOutliers API to compare GPU utilization, memory usage, temperature, power draw, and SM clock frequency within a training job. Static rules then identify XID errors, ECC memory faults, accelerator failures, EFA throughput degradation, packet drops and retransmissions, thermal or reliability throttling, and overheating.

EKS clusters install the amazon-sagemaker-hyperpod-observability add-on, while Slurm clusters use the sagemaker-observability extension. Quick setup enables observability by default and does not allow it to be disabled; Custom setup lets operators select metric categories. IAM controls who can install or modify the add-on or extension, preserving the cluster's existing administrative governance model.

How It Works: From Fleet Averages to a Specific Rank

The service collects three related telemetry layers:

| Layer | Key signals | Engineering purpose | |---|---|---| | Host and job | Cluster health, CPU, disk, OS, per-job resources | Detect data-pipeline stalls, host contention, and job-level anomalies | | GPU and Neuron | Utilization, memory, temperature, clocks, power, PCIe activity, DCGM and error counters | Detect degraded devices, power constraints, hardware faults, and thermal throttling | | Network and collective communication | EFA throughput, drops, retransmissions, errors, NCCL All-Reduce latency | Separate network congestion and retransmission delays from accelerator faults |

Synchronous distributed training is particularly sensitive to a slow rank. A step can be approximated as T_step ≈ max_i(t_compute,i) + T_collective: when one rank slows down, the remaining ranks wait at the collective barrier. In a ring All-Reduce, communication volume per rank is approximately V_rank ≈ 2(N−1)S/N, where S is the tensor size and N is the number of participating processes. Collective time then depends on effective bandwidth: T_collective ≈ V/B_eff + latency. EFA packet loss and retransmission do not merely reduce nominal throughput; they increase tail latency. Correlating EFA counters, NCCL latency, and accelerator compute metrics is therefore much more actionable than inspecting fleet-wide GPU utilization alone.

The diagnostic pipeline applies two complementary methods. DetectGroupOutliers finds statistical asymmetry: one GPU's utilization, memory, temperature, power, or SM clock differs significantly from its peers in the same job. Static rules map recognized failure signatures into hardware, network, throttling, or overheating incidents, then attach the incident to the affected job, cluster, node, accelerator, and rank. The first method can expose behavior not covered by predefined rules; the second explains which known failure class best fits the evidence.

Two sources of false positives remain. First, GPUs of different types within one job, or ranks assigned different roles such as input loading, checkpointing, or evaluation, may have legitimately different distributions. Startup, warm-up, checkpoint-saving, and shutdown phases can also create short-lived outliers. Second, static rules only explain failure modes already encoded in the service. AWS does not disclose the peer-grouping rules, statistical windows, thresholds, or confidence levels for outlier detection. Production teams should therefore suppress startup and checkpoint windows where possible and validate false-positive boundaries at the alerting layer.

When Managed Prometheus for Grafana is selected as the metric store, the Grafana visualization path is explicit. AWS does not fully explain how DetectGroupOutliers obtains the same data and triggers the static rules in that configuration, so teams should validate diagnostic-summary parity before adopting it.

Engineering Trade-offs: Visibility, Cardinality, and Cost

AWS reports approximately 1–2% CPU overhead and 250–300 MB of memory consumption for default collection. The post does not clarify whether this is measured per node, per collector, or across the deployment, so it should not be multiplied directly by fleet size to estimate thousand-GPU overhead. Advanced metrics add further series, labels, ingestion volume, storage, and alarm-evaluation work.

Metric cardinality is the largest hidden cost. As an illustrative calculation, 1,024 GPUs exporting 100 independent series each at 30-second intervals would generate about 295 million sample points per day, excluding host and network telemetry. Labels for node, job, rank, accelerator, and owning team can split the same underlying measurement into even more queryable series. AWS says the default CloudWatch option is less expensive than Managed Prometheus, but high cardinality can increase the bill on either backend.

A 30-second interval is appropriate for sustained degradation but may smooth brief power spikes or short congestion events. Moving to a finer interval should be justified by the target failure duration and ingestion budget rather than applied automatically.

CloudWatch is now the default metric store and integrates natively with SageMaker Insights. Teams already standardized on Prometheus and Grafana can retain Managed Prometheus. First-time Amazon Managed Grafana users must configure AWS IAM Identity Center and add Grafana users. Beyond installation permissions, production environments should also restrict dashboard queries, log access, and long-term metric retention so that training topology, job names, and ownership metadata are not exposed unnecessarily.

Performance Evidence and Evaluation Limits

The main quantitative disclosure concerns collection overhead. AWS has not published anomaly-detection precision, false-positive rates, alerts per GPU per day, time from degradation to alert, time to acknowledgement, maximum supported fleet size, or thousand-GPU ingestion cost. Node counts, accelerator counts, and utilization values shown in the interface are examples rather than performance benchmarks. The claim that diagnosis becomes faster is therefore a plausible operational benefit, not a publicly validated benchmark result.

A useful enterprise acceptance test should track:

  • Detection latency from a single-GPU slowdown or EFA degradation to an alert;
  • False positives per GPU-day and the proportion of actionable alerts;
  • Success in mapping an alert to the correct job, node, accelerator, and rank;
  • Telemetry samples, ingestion cost, and storage cost per GPU-day;
  • Missing-metric rate, collection delay, and alarm-rule evaluation delay.

Deployment Playbook

  1. Start with baseline telemetry: Use Custom setup to retain the default metric set for at least one complete training cycle before enabling advanced metrics. Quick setup cannot disable observability and is unsuitable for clusters that require strict collection policies.
  2. Enable advanced metrics by failure surface: Prioritize DCGM or Neuron metrics for the accelerator types in use, plus EFA metrics for clusters that actually rely on high-speed interconnect. There is little reason to collect every advanced series for small jobs or clusters without EFA.
  3. Standardize the metric backend deliberately: Use CloudWatch when the team needs native SageMaker Insights integration; use Managed Prometheus when an existing Prometheus and Grafana operating model is more important. Validate the diagnostic-summary path separately for Prometheus deployments.
  4. Create event-specific runbooks: Map XID or ECC findings to node-isolation and device-replacement procedures, EFA degradation to fabric and NCCL All-Reduce checks, and thermal throttling to job migration and power-policy reviews. A diagnostic-summary card is evidence for investigation, not automatic proof of physical device failure.
  5. Rehearse failures in isolation: Validate detection latency, notification routing, and recovery with controlled node slowdown, packet loss, or throttling in a test cluster. Do not intentionally induce XID or ECC faults in a production training job.
  6. Treat cost as an operational SLO: Count samples by metric category, restrict unused high-cardinality dimensions, and calculate ingestion, storage, query, and alerting costs separately for CloudWatch and Prometheus.

The update does not alter model architecture or training algorithms. It addresses a different failure surface in large-scale AI systems: the relationship between compute hardware, interconnect fabric, and distributed processes. On fleets where one degraded accelerator can stall an entire job, turning fleet averages into a precise job-and-rank diagnosis is production-critical infrastructure rather than a cosmetic observability feature.


Source: AWS Machine Learning Blog (2026-11-20)

— END —

Tags: #SageMaker HyperPod #GPU Observability #Distributed Training #CloudWatch #EFA

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…