Large distributed training jobs often fail due to network issues or hardware errors.
A single GPU fault can cause timeouts that crash healthy workers and waste expensive resources.
Synchronous checkpoints also block all ranks during saves, consuming significant time.
NVIDIA Resiliency Extension (NVRx) integrates with PyTorch Fully Sharded Data Parallel (FSDP) on Amazon EKS.
Users install NVRx via pip to add fault tolerance without changing their training code.
Three features help: async checkpointing, in-process restart, and automatic worker respawn.
Async checkpointing uses a background process to save state while training continues.
In-process restart catches transient faults like unhandled exceptions or NCCL hangs.
The ft_launcher binary handles hard crashes such as SIGKILL or out-of-memory kills.
Tests ran on 8 NVIDIA H100 GPUs across 2 to 8 nodes using Amazon EKS.
Nodes use p5.48xlarge instances with 32 Elastic Fabric Adapter (EFA) network interfaces.
Training pods run as Kubernetes Jobs with headless Services for peer discovery.
Amazon FSx for Lustre provides shared storage for checkpoint persistence in the same Availability Zone.
Recovering workers read checkpoints from this shared filesystem to resume training quickly.
