Amazon SageMaker HyperPod Adds Training Validation and Hung-Job Recovery
Summary:Amazon SageMaker HyperPod now adds three layers of workload recovery. On EKS, MultiNodeTrainingValidation runs fixed Megatron-LM workloads to detect GPUs that pass hardware diagnostics but produce NaN or infinite loss. The HyperPod Kubernetes Resource Monitoring Agent correlates node, device-plugin, DaemonSet, training-job, and optional NCCL RAS conditions with Kubernetes events and structured CloudWatch logs. For Slurm clusters, hung-job detection combines stdout freshness with GPU activity or
From Node Health to Workload Health
Amazon SageMaker HyperPod’s new capabilities address a critical gap in distributed training: GPUs, NVLinks, and EFAs can all pass diagnostics while a job still produces invalid values, stalls at a collective barrier, or loses the ability to progress after a device-plugin failure. Existing HMA, InstanceStress, and InstanceConnectivity checks mainly answer whether the hardware is healthy. The new capabilities go further: can this hardware train correctly, is the job still advancing, and which Pod and node owns the fault?
| Layer | Representative mechanism | Failures detected | Automatic remediation | | --- | --- | --- | --- | | Node hardware | HMA, InstanceStress, DCGM Level 4 | GPU, memory, NVLink, and EFA faults | Nodes may be rebooted or replaced | | Inter-node fabric | InstanceConnectivity, NCCL/NCCOM tests | Slow or broken communication | Primarily used as an admission check | | Training correctness | MultiNodeTrainingValidation | Hardware that yields NaN, infinite loss, or incomplete training | Failed instances can be labeled and replaced | | Kubernetes observability | Resource Monitoring Agent | Device-plugin crashes, missing DaemonSets, degraded nodes, failed jobs | Reports only; it does not restart workloads | | Job liveness | Slurm Hung Job Detection | Jobs still marked Running but no longer progressing | Can notify or cancel for Slurm requeue |
MultiNodeTrainingValidation: From Eligible Hardware to Correct Computation
MultiNodeTrainingValidation is a new deep health check for EKS-based HyperPod clusters. It uses two fixed reference workloads: a Tiny Megatron test across every GPU in one instance, followed by a multi-instance Megatron test between instances of the same type over EFA. An instance fails if the loss becomes NaN or infinite, or if training does not finish within the configured limit.
The check targets silent data corruption that ordinary stress tests can miss. A GPU’s memory, compute units, and fabric may all report healthy while a particular matrix operation, reduction, or cross-node collective still returns incorrect values. Distributed training amplifies local faults: if one rank writes NaN, gradient synchronization can contaminate other ranks; if one rank falls behind, an all_reduce, barrier, or rendezvous can hold the entire process group in place. MultiNodeTrainingValidation uses a fixed, known-good training path to verify end-to-end computational correctness before a production job enters the queue.
Results are written to CloudWatch and exposed through node labels. With automatic node recovery enabled, HyperPod can reboot or replace a failed instance. The boundary is equally important: passing a reference workload does not guarantee convergence for every model, batch size, or parallelism strategy. The feature currently targets supported EFA-enabled NVIDIA GPU instances, and a fixed model cannot cover every possible failure mode.
Kubernetes Resource Monitoring Agent: Restoring the Control-Plane-to-Job Causal Chain
The new Resource Monitoring Agent runs as one Pod per cluster and prefers CPU nodes so it does not consume training GPU capacity. It continuously observes:
- Node conditions, allocatable GPU and EFA counts, NVIDIA capability labels, and taint changes;
- Critical DaemonSets, device plugins, and training controllers in
kube-system,aws-hyperpod, andkubeflow; - Failure reasons for HyperPodPyTorchJob, Kubeflow PyTorchJob v1, and Kubeflow TrainJob v2;
- Optional NCCL RAS communication state.
When it finds a problem, the agent correlates it with the affected node, Pod, DaemonSet, and training job, then writes both Kubernetes Events and structured CloudWatch JSON logs. Events support immediate triage but expire; CloudWatch retains a longer investigation history. NCCL 2.24 and later can expose a RAS listener in each NCCL process. Every 30 seconds, the agent reads the per-Pod reports: valid reports from every Pod mean healthy, partial validity means degraded, and no valid reports means unhealthy. In one test, a rank stopped invoking all_reduce. The system detected unequal collective operation counts across ranks and identified the lagging GPU, process, and node.
The agent deliberately observes rather than remediates. It does not reboot nodes or restart jobs, avoiding destructive recovery before the root cause is understood, but this also means it improves diagnosis rather than providing end-to-end self-healing. Enabling NCCL RAS requires binding its listener to the Pod IP and exposing TCP port 28028. A NetworkPolicy should restrict that port to the monitoring agent and the job’s own Pods.
Slurm Hung-Job Detection: Moving Beyond the Running State
For HyperPod Slurm clusters, AWS has added hung-job detection that requires no changes to training scripts or containers. Administrators point HYPERPOD_HANG_RULES to a rules file readable from every compute node.
Default detection combines Slurm output freshness with GPU activity:
- When output stops while GPUs are idle but still hold memory, it can identify rendezvous or barrier hangs within a shorter window;
- When output stops for longer, it can catch NCCL timeouts, stalled data pipelines, and deadlocks;
- Custom
log_patternrules can require a training step or epoch to appear within a deadline, or immediately flag a matching NCCL error.
A detected hang produces a HANG_DETECTED event in CloudWatch. In notify mode the system only alerts; in cancel mode it cancels the job so Slurm can requeue it under the cluster’s MaxBatchRequeue limit. Custom rules replace default detection for that job rather than simply augmenting it. Default detection can also miss a job whose GPUs remain busy inside a useless loop. Production users should therefore begin in notify mode, calibrate for normal checkpoint and evaluation gaps, and only then enable automatic cancellation.
The rules file is part of the control plane. Anyone able to modify or replace it can change detection behavior or cause the agent to cancel every job using that file. A personal file should use 600 permissions; a shared file needs a protected parent directory and a clearly designated maintainer. A rules file supports up to 20 rules and a maximum size of 1 MB. Existing HyperPod Training Operator and Ray Train hang detection remain available for EKS; this launch fills the Slurm path.
Measured Results, Cost Model, and Evaluation Limits
AWS reports an internal test on two ml.p5.48xlarge instances, totaling 16 H100 GPUs. The full deep health check took 1 hour and 28 minutes, with DCGM Level 4 diagnostics accounting for 74 minutes. The new Tiny Megatron and multi-instance Megatron tests added about 8 minutes. The multi-instance test completed 300 iterations, reduced loss from 11.5671 to 3.6213, encountered no NaN or skipped iterations, and finished in 161.5 seconds, or about 1.86 iterations per second.
These results demonstrate stable convergence of the reference workload on one configuration, but they are not an independent benchmark. The test covers a single two-node setup and provides no fleet-wide detection rate, false-positive rate, confidence interval, or results across models, batch sizes, topologies, and injected faults. Two costs deserve attention: MultiNodeTrainingValidation keeps instances unavailable for several additional minutes, while overly strict loss thresholds or timeouts can label healthy hardware as faulty.
The source also provides a modeled failure cost. A job spanning 32 ml.p5.48xlarge instances, or 256 H100 GPUs, hangs at 1:00 AM and is noticed at 9:00 AM. It wastes 256 instance-hours. At approximately $66 per instance-hour in US East, the loss is about $16,900. A 15-minute rule that cancels and requeues the job limits the cost to roughly $530, saving about $16,400. This is a reasonable scenario model, not an average saving measured through a production A/B test.
Deployment Guidance for EKS Clusters
- Add
MultiNodeTrainingValidationtoOnStartDeepHealthChecksin the instance-group configuration while retainingInstanceStressandInstanceConnectivityfor hardware, fabric, and training-correctness coverage. - Before a long pre-training run, use
StartClusterHealthCheckto revalidate existing instances and narrow the scope withInstanceIdswhere appropriate. - Apply the Resource Monitoring Agent CRD manually before upgrading the HyperPod Helm Chart, then enable
hyperpod-kubernetes-resource-monitor. Use a server-side dry run first because the upgrade also moves HMA and device plugins to the chart’s current versions. - Explicitly list the required NVIDIA and EFA device-plugin DaemonSets and all business-job namespaces. These lists replace defaults, so omitting HMA or a workload namespace creates blind spots.
- For NCCL RAS, deploy a NetworkPolicy allowing port 28028 only from the agent and job Pods before setting
NCCL_RAS_ADDRto the Pod IP. - Route Events such as
TrainingJobFailed,CriticalPodCrashLoop,NecessaryDaemonSetMissing, andNodeResourceDegradedinto alerts, while retaining structured CloudWatch logs for post-incident analysis.
Deployment Guidance for Slurm Clusters
- Use a HyperPod Slurm AMI released on or after November 10, 2026. Software updates replace the instance root volume, so back up local data first. An existing cluster can be checked by seeing whether
srun --helplists--no-hang-detection. - Place the rules file on shared storage such as FSx for Lustre or EFS and keep the same path on every compute node.
- First enable default detection with
{"enabled": true}, pointHYPERPOD_HANG_RULESat the file for a test job, and verify that the expected CloudWatch events appear. - Add progress rules for training steps, epochs, or checkpoints, with timeouts longer than normal model loading, compilation, checkpoint-save, and evaluation intervals.
- Run in notify mode to collect false-positive evidence before switching to cancel, and inspect Slurm’s
MaxBatchRequeueto prevent repeated requeues from amplifying a fault. - Create a CloudWatch metric filter and SNS alarm for
HANG_DETECTED, and alert onCONFIG_ERRORso a missing or invalid rules file cannot silently disable monitoring.
Engineering Assessment
This update does not alter the underlying Megatron, NCCL, or Kubernetes protocols. Instead, it completes a platform-level loop spanning hardware admission, computational correctness, control-plane observability, and job-liveness detection. Its main contribution is not that it automatically repairs every fault, but that it moves silent corruption, collective stalls, and infrastructure failures into an earlier detection window and correlates them with specific nodes, Pods, GPUs, and processes.
The system remains layered rather than fully autonomous. Training validation proves only that a reference workload succeeds; the Kubernetes agent reports but does not recover; hung-job detection depends on log semantics and timeout calibration. Real production gains will depend on false-positive rates, checkpoint strategy after requeue, rules-file permissions, topology coverage, and CloudWatch alert quality. For organizations running thousand- to ten-thousand-GPU pre-training, these features can reduce expensive idle allocations, but fault injection and real-job log validation remain necessary before treating them as fully proven reliability improvements.
— END —Source: AWS Machine Learning Blog (2026-11-18)
Tags: #Amazon SageMaker HyperPod #Distributed Training #GPU Clusters #Fault Recovery #Kubernetes
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments