explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR: what shipped
  • What problem does it solve?
  • How does the diagnostic summary work?
  • Default and advanced metrics
  • CloudWatch becomes the default store
  • How do I turn it on?
  • What this means for teams training models
  • A triage routine you can apply to any slow training job
  • How it compares with doing it yourself
  • Questions teams will ask
  • Why this matters beyond AWS
  • Limits and open questions
← Back to blog

explainx / blog

SageMaker HyperPod Enhanced Observability: Find the Bad GPU in a Training Job

AWS, SageMaker HyperPod, AI Infrastructure, GPU Training, Observability

Part of Microsoft, Apple and Amazon AI

AWS added enhanced observability to SageMaker HyperPod: GPU outlier detection, XID and EFA rules, CloudWatch as default store. What it does and how to enable it.

Nov 19, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
SageMaker HyperPod Enhanced Observability: Find the Bad GPU in a Training Job

Amazon announced enhanced observability for SageMaker HyperPod on November 19, 2026. It brings diagnostic and profiling tools together for training jobs on HyperPod clusters orchestrated by Amazon EKS or Slurm. AWS frames the problem plainly in its announcement post: noticing that training slowed is easy, and working out whether a throttled GPU, a saturated network link or a stalled host data pipeline caused it is the costly part.

TL;DR: what shipped

table · 2 cols
QuestionAnswer (per AWS)
Where does it run?HyperPod on EKS (add-on) and Slurm (extension)
Main featureDiagnostic summary: GPU outlier detection plus static failure rules
Outlier metricsUtilization, memory, temperature, power, SM clock
Failure rulesHardware (XID, ECC), network (EFA), throttling, overheating
Default metric storeAmazon CloudWatch; Amazon Managed Service for Prometheus still supported
Default overheadAbout 1 to 2 percent CPU, 250 to 300 MB memory, 30-second interval
Opt-inAdvanced metrics: node exporter suite, DCGM and Neuron, EFA
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

What problem does it solve?

Large-scale training is one of the most expensive GPU workloads, and AWS notes a single unhealthy accelerator can stall an entire distributed job. Until now, AWS says, visibility into HyperPod meant stitching separate tools together, and the metrics showed that performance had degraded without saying why. Enhanced observability targets three gaps, the first being isolating one bad accelerator among hundreds or thousands, whether through XID errors, ECC faults or thermal throttling.

If you have read our coverage of NVIDIA's open-source Attribution tool for failed training jobs or Recovery for crashed workloads, this is the managed-service counterpart: AWS built the diagnosis into the platform.

How does the diagnostic summary work?

Two approaches run together.

GPU outlier detection calls CloudWatch's DetectGroupOutliers API to compare GPUs within one training job. A GPU that deviates significantly from its peers appears as an outlier card. It works for EKS and Slurm jobs. The peer comparison matters because in synchronous training all GPUs should behave alike, so the odd one out is the suspect.

Static diagnostic rules check known failure patterns, grouped by type: hardware (ECC memory errors, XID faults, accelerator failures), network (EFA throughput degradation, drops, retransmissions), throttling (clock limits from heat, board or reliability) and overheating (temperatures above configured limits). Each issue names the affected job, cluster, node, accelerator and rank.

Default and advanced metrics

Default metrics are on automatically. They cover cluster health, CPU, disk and OS usage, GPU utilization and temperature, and per-job resource use, collected every 30 seconds with the overhead above. Advanced metrics are opt-in: the full node exporter suite, all NVIDIA DCGM and AWS Neuron metrics (clocks, power, PCIe activity, error indicators) and Elastic Fabric Adapter metrics (throughput, drops, retransmissions, errors).

AWS says advanced metrics have a cost implication because they increase telemetry volume. Its guidance: small jobs on small clusters probably do not need them, while long-running jobs on large clusters do, because patterns emerge over time.

CloudWatch becomes the default store

Previously only Prometheus was supported. CloudWatch is now the default, with native views through CloudWatch SageMaker Insights, lower pricing than Amazon Managed Service for Prometheus according to AWS, and tighter console integration. Teams on Prometheus and Grafana can keep that route. We have not verified the price difference; check the AWS pricing pages for your region and metric volume.

How do I turn it on?

  1. Create a HyperPod cluster or use an existing one (console: HyperPod clusters, then Create, then EKS or Slurm orchestration with custom setup).
  2. Install the observability add-on (EKS) or extension (Slurm) from the console, the API or the CLI.
  3. For EKS, AWS's example runs aws eks create-addon with the add-on name amazon-sagemaker-hyperpod-observability, the metrics destination set to cloudwatch, basic node and accelerated compute metrics at a 30-second scrape interval, and host profiling off.
  4. Check the add-on status is ACTIVE with aws eks describe-addon.
  5. Enable more categories later with aws eks update-addon, for example setting network metrics to ADVANCED to get EFA data, or use Actions then Edit in the console.

Because it ships as an add-on or extension, you control who installs or changes it through IAM. Copy exact configuration from the AWS post, since the values are specific to its example.

What this means for teams training models

If you run long jobs on HyperPod, the practical gain is time to diagnosis. Instead of correlating logs with dashboards, you start from the flagged GPU and rank. The cost of a stalled job is the whole cluster idling, so catching one throttled card early is the saving AWS is selling. For agent and inference workloads, the same AWS line of work appears in SageMaker KV-cache-aware routing and full fine-tuning in serverless customization.

A triage routine you can apply to any slow training job

AWS does not publish a runbook in the post, so the following is our suggested way to use the three signal types it describes. It is guidance, not an AWS procedure.

  1. Start with the outlier cards. If one GPU deviates on temperature, power or SM clock, note its node, accelerator and rank. A single deviating card points to a hardware or cooling cause on that node rather than to your code.
  2. Read the rule category. Hardware findings (XID or ECC) usually mean you should cordon the node and replace or reboot it. Throttling findings mean the card is healthy but limited by heat, board or reliability limits, so look at airflow and neighbors. Overheating findings cross a configured temperature limit.
  3. Check the network rules next. If every GPU looks normal but throughput dropped, EFA throughput degradation, drops or retransmissions suggest a congested or faulty link. Those need the advanced EFA metrics, so enable them before a long run, not after the slowdown.
  4. Only then suspect the host. The post names a stalled data pipeline on the host as a third cause. Host-level metrics from the default set, and the opt-in host profiling, are where that shows up.
  5. Write down the rank. Because each issue names job, cluster, node, accelerator and rank, you can map it to a specific worker in your distributed setup and restart from a checkpoint without that node.

The value of the order is that it moves from the cheapest signal (peer comparison, always on) to the most expensive (advanced metrics and profiling). That also matches AWS's cost note: leave advanced metrics off on small jobs and turn them on for long runs on large clusters.

How it compares with doing it yourself

Before a managed feature like this, teams commonly combine DCGM exporters, node exporters, a Prometheus and Grafana stack, and custom alert rules. That works and remains supported here, since Amazon Managed Service for Prometheus is still an option. What AWS adds is the packaging: a pre-built diagnostic summary, rules for named failure patterns, and an add-on lifecycle controlled with IAM. What it does not add, according to the post, is automatic repair. The post describes diagnosis and profiling, not replacing a node. For automated restart behavior in Slurm environments, see NVIDIA's open-source Recovery tool linked above.

Questions teams will ask

Does it work on both orchestrators? Yes, AWS says the capabilities apply to EKS and Slurm clusters, delivered as an add-on and an extension respectively.

Can I change settings later? AWS says you can enable or disable any metric category or profiling capability at any time, from the CLI or from the add-on card in the console.

Will the default settings slow my job? AWS quotes roughly 1 to 2 percent CPU overhead and 250 to 300 MB of memory for default metrics. Measure on your own cluster, because those figures are the vendor's.

Is this only for NVIDIA GPUs? The post mentions both NVIDIA GPU DCGM metrics and AWS Neuron metrics among the advanced set, so accelerators from both families are covered by the metric lists. Check the HyperPod documentation for the exact support on your instance type.

Where do I start reading? The AWS Machine Learning blog post has the console steps, the CLI commands and the screenshots of the add-on tab.

Why this matters beyond AWS

Training reliability is becoming a competitive topic as labs run ever larger clusters. When a run spans thousands of accelerators, the odds that at least one card is degraded at any moment rise sharply, so tooling that names the failing rank quickly is not a luxury. Cloud vendors are therefore folding diagnosis into the platform instead of leaving it to each customer. If you evaluate training platforms, ask each vendor three things: how it finds the single slow GPU, how much telemetry overhead it adds, and whether diagnosis connects to automatic recovery.

Limits and open questions

  • The overhead figures and the 30-second interval are AWS's own numbers; we have not benchmarked them.
  • AWS gives no detection accuracy or false-positive rate for outlier flagging.
  • Outlier detection compares GPUs within a job, so a fault affecting every GPU the same way would not stand out.
  • Advanced metrics raise cost; AWS does not publish a worked example in the post we read.

Related reading on explainx.ai:

  • NVIDIA Attribution for failed training jobs
  • NVIDIA Recovery for training workloads
  • SageMaker KV-cache-aware routing
  • SageMaker full fine-tuning, serverless
  • NVIDIA CUDA 13.4 GPU sharing on Kubernetes
  • AWS Bedrock AgentCore runtime

Details are accurate as of November 19, 2026 and come from AWS.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 19, 2026

SageMaker Cross-Region Inference Routing: Failover Without Writing Your Own Router

AWS made cross-Region inference routing generally available in SageMaker AI on November 19, 2026. A new inference router sends requests to a healthy endpoint in another Region or account when the local one runs short of GPUs. Here is how it decides, what it costs and where it stops.

Nov 17, 2026

Amazon $17B Ohio AWS Data Centers: What Voter Bans Put at Risk

Ohio already passed the country densest data-center ballot season. A November 17 Financial Times analysis adds a dollar figure: more than $17 billion in Amazon Web Services spend across Ohio is now at risk where three proposed campuses hit new local limits.

Nov 17, 2026

Blackstone BXDC $5B Amazon Data Centers: Cap Rate vs 50% Cheaper Headline

Blackstone’s newly listed BXDC signed a $5 billion deal for majority stakes in 13 Amazon Web Services data-center shells in Indiana and Mississippi. Official documents describe a 7.0% initial cap rate and a 12-year AWS lease-back—not a 50% discount versus new construction.