Amazon announced enhanced observability for SageMaker HyperPod on November 19, 2026. It brings diagnostic and profiling tools together for training jobs on HyperPod clusters orchestrated by Amazon EKS or Slurm. AWS frames the problem plainly in its announcement post: noticing that training slowed is easy, and working out whether a throttled GPU, a saturated network link or a stalled host data pipeline caused it is the costly part.
TL;DR: what shipped
| Question | Answer (per AWS) |
|---|---|
| Where does it run? | HyperPod on EKS (add-on) and Slurm (extension) |
| Main feature | Diagnostic summary: GPU outlier detection plus static failure rules |
| Outlier metrics | Utilization, memory, temperature, power, SM clock |
| Failure rules | Hardware (XID, ECC), network (EFA), throttling, overheating |
| Default metric store | Amazon CloudWatch; Amazon Managed Service for Prometheus still supported |
| Default overhead | About 1 to 2 percent CPU, 250 to 300 MB memory, 30-second interval |
| Opt-in | Advanced metrics: node exporter suite, DCGM and Neuron, EFA |
What problem does it solve?
Large-scale training is one of the most expensive GPU workloads, and AWS notes a single unhealthy accelerator can stall an entire distributed job. Until now, AWS says, visibility into HyperPod meant stitching separate tools together, and the metrics showed that performance had degraded without saying why. Enhanced observability targets three gaps, the first being isolating one bad accelerator among hundreds or thousands, whether through XID errors, ECC faults or thermal throttling.
If you have read our coverage of NVIDIA's open-source Attribution tool for failed training jobs or Recovery for crashed workloads, this is the managed-service counterpart: AWS built the diagnosis into the platform.
How does the diagnostic summary work?
Two approaches run together.
GPU outlier detection calls CloudWatch's DetectGroupOutliers API to compare GPUs within one training job. A GPU that deviates significantly from its peers appears as an outlier card. It works for EKS and Slurm jobs. The peer comparison matters because in synchronous training all GPUs should behave alike, so the odd one out is the suspect.
Static diagnostic rules check known failure patterns, grouped by type: hardware (ECC memory errors, XID faults, accelerator failures), network (EFA throughput degradation, drops, retransmissions), throttling (clock limits from heat, board or reliability) and overheating (temperatures above configured limits). Each issue names the affected job, cluster, node, accelerator and rank.
Default and advanced metrics
Default metrics are on automatically. They cover cluster health, CPU, disk and OS usage, GPU utilization and temperature, and per-job resource use, collected every 30 seconds with the overhead above. Advanced metrics are opt-in: the full node exporter suite, all NVIDIA DCGM and AWS Neuron metrics (clocks, power, PCIe activity, error indicators) and Elastic Fabric Adapter metrics (throughput, drops, retransmissions, errors).
AWS says advanced metrics have a cost implication because they increase telemetry volume. Its guidance: small jobs on small clusters probably do not need them, while long-running jobs on large clusters do, because patterns emerge over time.
CloudWatch becomes the default store
Previously only Prometheus was supported. CloudWatch is now the default, with native views through CloudWatch SageMaker Insights, lower pricing than Amazon Managed Service for Prometheus according to AWS, and tighter console integration. Teams on Prometheus and Grafana can keep that route. We have not verified the price difference; check the AWS pricing pages for your region and metric volume.
How do I turn it on?
- Create a HyperPod cluster or use an existing one (console: HyperPod clusters, then Create, then EKS or Slurm orchestration with custom setup).
- Install the observability add-on (EKS) or extension (Slurm) from the console, the API or the CLI.
- For EKS, AWS's example runs
aws eks create-addonwith the add-on nameamazon-sagemaker-hyperpod-observability, the metrics destination set to cloudwatch, basic node and accelerated compute metrics at a 30-second scrape interval, and host profiling off. - Check the add-on status is ACTIVE with
aws eks describe-addon. - Enable more categories later with
aws eks update-addon, for example setting network metrics to ADVANCED to get EFA data, or use Actions then Edit in the console.
Because it ships as an add-on or extension, you control who installs or changes it through IAM. Copy exact configuration from the AWS post, since the values are specific to its example.
What this means for teams training models
If you run long jobs on HyperPod, the practical gain is time to diagnosis. Instead of correlating logs with dashboards, you start from the flagged GPU and rank. The cost of a stalled job is the whole cluster idling, so catching one throttled card early is the saving AWS is selling. For agent and inference workloads, the same AWS line of work appears in SageMaker KV-cache-aware routing and full fine-tuning in serverless customization.
A triage routine you can apply to any slow training job
AWS does not publish a runbook in the post, so the following is our suggested way to use the three signal types it describes. It is guidance, not an AWS procedure.
- Start with the outlier cards. If one GPU deviates on temperature, power or SM clock, note its node, accelerator and rank. A single deviating card points to a hardware or cooling cause on that node rather than to your code.
- Read the rule category. Hardware findings (XID or ECC) usually mean you should cordon the node and replace or reboot it. Throttling findings mean the card is healthy but limited by heat, board or reliability limits, so look at airflow and neighbors. Overheating findings cross a configured temperature limit.
- Check the network rules next. If every GPU looks normal but throughput dropped, EFA throughput degradation, drops or retransmissions suggest a congested or faulty link. Those need the advanced EFA metrics, so enable them before a long run, not after the slowdown.
- Only then suspect the host. The post names a stalled data pipeline on the host as a third cause. Host-level metrics from the default set, and the opt-in host profiling, are where that shows up.
- Write down the rank. Because each issue names job, cluster, node, accelerator and rank, you can map it to a specific worker in your distributed setup and restart from a checkpoint without that node.
The value of the order is that it moves from the cheapest signal (peer comparison, always on) to the most expensive (advanced metrics and profiling). That also matches AWS's cost note: leave advanced metrics off on small jobs and turn them on for long runs on large clusters.
How it compares with doing it yourself
Before a managed feature like this, teams commonly combine DCGM exporters, node exporters, a Prometheus and Grafana stack, and custom alert rules. That works and remains supported here, since Amazon Managed Service for Prometheus is still an option. What AWS adds is the packaging: a pre-built diagnostic summary, rules for named failure patterns, and an add-on lifecycle controlled with IAM. What it does not add, according to the post, is automatic repair. The post describes diagnosis and profiling, not replacing a node. For automated restart behavior in Slurm environments, see NVIDIA's open-source Recovery tool linked above.
Questions teams will ask
Does it work on both orchestrators? Yes, AWS says the capabilities apply to EKS and Slurm clusters, delivered as an add-on and an extension respectively.
Can I change settings later? AWS says you can enable or disable any metric category or profiling capability at any time, from the CLI or from the add-on card in the console.
Will the default settings slow my job? AWS quotes roughly 1 to 2 percent CPU overhead and 250 to 300 MB of memory for default metrics. Measure on your own cluster, because those figures are the vendor's.
Is this only for NVIDIA GPUs? The post mentions both NVIDIA GPU DCGM metrics and AWS Neuron metrics among the advanced set, so accelerators from both families are covered by the metric lists. Check the HyperPod documentation for the exact support on your instance type.
Where do I start reading? The AWS Machine Learning blog post has the console steps, the CLI commands and the screenshots of the add-on tab.
Why this matters beyond AWS
Training reliability is becoming a competitive topic as labs run ever larger clusters. When a run spans thousands of accelerators, the odds that at least one card is degraded at any moment rise sharply, so tooling that names the failing rank quickly is not a luxury. Cloud vendors are therefore folding diagnosis into the platform instead of leaving it to each customer. If you evaluate training platforms, ask each vendor three things: how it finds the single slow GPU, how much telemetry overhead it adds, and whether diagnosis connects to automatic recovery.
Limits and open questions
- The overhead figures and the 30-second interval are AWS's own numbers; we have not benchmarked them.
- AWS gives no detection accuracy or false-positive rate for outlier flagging.
- Outlier detection compares GPUs within a job, so a fault affecting every GPU the same way would not stand out.
- Advanced metrics raise cost; AWS does not publish a worked example in the post we read.
Related reading on explainx.ai:
- NVIDIA Attribution for failed training jobs
- NVIDIA Recovery for training workloads
- SageMaker KV-cache-aware routing
- SageMaker full fine-tuning, serverless
- NVIDIA CUDA 13.4 GPU sharing on Kubernetes
- AWS Bedrock AgentCore runtime
Details are accurate as of November 19, 2026 and come from AWS.
