Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod
Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.
阅读原文 · AWS 机器学习博客本文由 SoccerDream AI 自动聚合索引,内容版权归原作者与 AWS 机器学习博客 所有。
本站仅作技术动态聚合,不构成任何形式的建议。
本站仅作技术动态聚合,不构成任何形式的建议。