In 2024, the Ministry of Economic system, Commerce and Trade (METI) launched the Generative AI Accelerator Problem (GENIAC)—a Japanese nationwide program to spice up generative AI by offering corporations with funding, mentorship, and big compute sources for basis mannequin (FM) growth. AWS was chosen because the cloud supplier for GENIAC’s second cycle (cycle 2). It offered infrastructure and technical steering for 12 taking part organizations. On paper, the problem appeared simple: give every crew entry to lots of of GPUs/Trainium chips and let innovation ensue. In apply, profitable FM coaching required way over uncooked {hardware}.
AWS found that allocating over 1,000 accelerators was merely the place to begin—the actual problem lay in architecting a dependable system and overcoming distributed coaching obstacles. Throughout GENIAC cycle 2, 12 prospects efficiently deployed 127 Amazon EC2 P5 cases (NVIDIA H100 TensorCore GPU servers) and 24 Amazon EC2 Trn1 cases (AWS Trainium1 servers) in a single day. Over the next 6 months, a number of large-scale fashions had been educated, together with notable tasks like Stockmark-2-100B-Instruct-beta, Llama 3.1 Shisa V2 405B, and Llama-3.1-Future-Code-Ja-8B, and others.
This submit shares the important thing insights from this engagement and beneficial classes for enterprises or nationwide initiatives aiming to construct FMs at scale.
Cross-functional engagement groups
A vital early lesson from technical engagement for the GENIAC was that operating a multi-organization, national-scale machine studying (ML) initiative requires coordinated help throughout various inner groups. AWS established a digital crew that introduced collectively account groups, specialist Options Architects, and repair groups. The GENIAC engagement mannequin thrives on shut collaboration between prospects and a multi-layered AWS crew construction, as illustrated within the following determine.
Clients (Cx) sometimes include enterprise and technical leads, together with ML and platform engineers, and are liable for executing coaching workloads. AWS account groups (Options Architects and Account Managers) handle the connection, keep documentation, and keep communication flows with prospects and inner specialists. The World Extensive Specialist Group (WWSO) Frameworks crew focuses on large-scale ML workloads, with a concentrate on core HPC and container providers comparable to AWS ParallelCluster, Amazon Elastic Kubernetes Service (Amazon EKS), and Amazon SageMaker HyperPod. The WWSO Frameworks crew is liable for establishing this engagement construction and supervising technical engagements on this program. They lead the engagement in partnership with different stakeholders and function an escalation level for different stakeholders. They work instantly with the service groups—Amazon Elastic Compute Cloud (Amazon EC2), Amazon Easy Storage Service (Amazon S3), Amazon FSx, and SageMaker HyperPod—to assist navigate engagements, escalations (enterprise and technical), and ensure the engagement framework is in working order. They supply steering on coaching and inference to prospects and educate different groups on the expertise. The WWSO Frameworks crew labored intently with Lead Options Architects (Lead SAs), a task particularly designated to help GENIAC engagements. These Lead SAs function a cornerstone of this engagement. They’re an extension of the Frameworks specialist crew and work instantly with prospects and the account groups. They interface with prospects and have interaction their Framework specialist counterparts when clarification or additional experience is required for in-depth technical discussions or troubleshooting. With this layered construction, AWS can scale technical steering successfully throughout complicated FM coaching workloads.
One other vital success issue for GENIAC was establishing sturdy communication channels between prospects and AWS members. The inspiration of our communication technique was a devoted inner Slack channel for GENIAC program coordination, connecting AWS account groups with lead SAs. This channel enabled real-time troubleshooting, information sharing, and fast escalation of buyer points to the suitable technical specialists and repair crew members. Complementing this was an exterior Slack channel that bridged AWS groups with prospects, making a collaborative surroundings the place individuals may ask questions, share insights, and obtain speedy help. This direct line of communication considerably decreased decision instances and fostered a neighborhood of apply amongst individuals.
AWS maintained complete workload monitoring paperwork, which clarifies every buyer’s coaching implementation particulars (mannequin structure, distributed coaching frameworks, and associated software program elements) alongside infrastructure specs (occasion varieties and portions, cluster configurations for AWS ParallelCluster or SageMaker HyperPod deployments, and storage options together with Amazon FSx for Lustre and Amazon S3). This monitoring system additionally maintained a chronological historical past of buyer interactions and help instances. As well as, the engagement crew held weekly evaluate conferences to trace excellent buyer inquiries and technical points. This common cadence made it doable for crew members to share classes discovered and apply them to their very own buyer engagements, fostering steady enchancment and information switch throughout this system.
With a structured strategy to communication and documentation, we may establish widespread challenges, comparable to misconfigured NCCL library impacting multi-node efficiency, share options throughout groups, and repeatedly refine our engagement mannequin. The detailed monitoring system offered beneficial insights for future GENIAC cycles, serving to us anticipate buyer wants and proactively tackle potential bottlenecks within the FM growth course of.
Reference architectures
One other early takeaway was the significance of strong reference architectures. Slightly than let every crew configure their very own cluster from scratch, AWS created pre-validated templates and automation for 2 predominant approaches: AWS ParallelCluster (for a user-managed HPC cluster) and SageMaker HyperPod (for a managed, resilient cluster service). These reference architectures coated the complete stack—from compute, community, and storage to container environments and monitoring—and had been delivered as a GitHub repository so groups may deploy them with minimal friction.
AWS ParallelCluster proved invaluable as an open supply cluster administration instrument for multi-node GPU coaching. AWS ParallelCluster automates the setup of a Slurm-based HPC cluster on AWS. AWS ParallelCluster simplifies cluster provisioning primarily based on the open supply Slurm scheduler, utilizing a easy YAML config to face up the surroundings. For the GEINIAC program, AWS additionally supplied SageMaker HyperPod as another choice for some groups. SageMaker HyperPod is a managed service that provisions GPU and Trainium clusters for large-scale ML. HyperPod integrates with orchestrators like Slurm or Kubernetes (Amazon EKS) for scheduling, offering further managed performance round cluster resiliency. By together with reference architectures for each AWS ParallelCluster and SageMaker HyperPod, the GENIAC program gave individuals flexibility—some opted for the fine-grained management of managing their very own HPC cluster, whereas others most popular the comfort and resilience of a managed SageMaker HyperPod cluster.
The reference structure (proven within the following diagram) seamlessly combines compute, networking, storage, and monitoring into an built-in system particularly designed for large-scale FM coaching.
The bottom infrastructure stack is offered as an AWS CloudFormation template that provisions the entire infrastructure stack with minimal effort. This template robotically configures a devoted digital non-public cloud (VPC) with optimized networking settings and implements a high-performance FSx for Lustre file system for coaching information (complemented by optionally available Amazon FSx for OpenZFS help for shared dwelling directories). The structure is accomplished with an S3 bucket that gives sturdy, long-term storage for datasets and mannequin checkpoints, sustaining information availability nicely past particular person coaching cycles. This reference structure employs a hierarchical storage strategy that balances efficiency and cost-effectiveness. It makes use of Amazon S3 for sturdy, long-term storage of coaching information and checkpoints, and hyperlinks this bucket to the Lustre file system by way of an information repository affiliation (DRA). The DRA permits computerized and clear information switch between Amazon S3 and FSx for Lustre, permitting high-performance entry with out guide copying. You should utilize the next CloudFormation template to create the S3 bucket used on this structure.
The optionally available monitoring infrastructure combines Amazon Managed Service for Prometheus and Amazon Managed Grafana (or self-managed Grafana service operating on Amazon EC2) to offer complete observability. It built-in DCGM Exporter for GPU metrics and EFA Exporter for community metrics, enabling real-time monitoring of system well being and efficiency. This setup permits for steady monitoring of GPU well being, community efficiency, and coaching progress, with automated alerting for anomalies by way of Grafana Dashboards. For instance, the GPU Well being Dashboard (see the next screenshot) gives metrics of widespread GPU errors, together with Uncorrectable Remapped Rows, Correctable Remapped Rows, XID Error Codes, Row Remap Failure, Thermal Violations, and Lacking GPUs (from Nvidia-SMI), serving to customers establish {hardware} failures as shortly as doable.
Reproducible deployment guides and structured enablement periods
Even the very best reference architectures are solely helpful if groups know how one can use them. A vital factor of GENIAC’s success was reproducible deployment guides and structured enablement by way of workshops.On October 3, 2024, AWS Japan and the WWSO Frameworks crew carried out a mass enablement session for GENIAC Cycle 2 individuals, inviting Frameworks crew members from america to share greatest practices for FM coaching on AWS.
The enablement session welcomed over 80 individuals and offered a complete mixture of lectures, hands-on labs, and group discussions—incomes a CSAT rating of 4.75, reflecting its robust influence and relevance to attendees. The lecture periods coated infrastructure fundamentals, exploring orchestration choices comparable to AWS ParallelCluster, Amazon EKS, and SageMaker HyperPod, together with the software program elements needed to construct and prepare large-scale FMs utilizing AWS. The periods highlighted sensible challenges in FM growth—together with large compute necessities, scalable networking, and high-throughput storage—and mapped them to applicable AWS providers and greatest practices. (For extra info, see the slide deck from the lecture session.) One other session centered on greatest practices, the place attendees discovered to arrange efficiency dashboards with Prometheus and Grafana, monitor EFA site visitors, and troubleshoot GPU failures utilizing NVIDIA’s DCGM toolkit and customized Grafana dashboards primarily based on the Frameworks crew’s expertise managing a cluster with 2,000 P5 cases.
Moreover, the WWSO crew ready workshops for each AWS ParallelCluster (Machine Studying on AWS ParallelCluster) and SageMaker HyperPod (Amazon SageMaker HyperPod Workshop), offering detailed deployment guides for the aforementioned reference structure. Utilizing these supplies, individuals carried out hands-on workout routines deploying their coaching clusters utilizing Slurm with file programs together with FSx for Lustre and FSx for OpenZFS, operating multi-node PyTorch distributed coaching. One other section of the workshop centered on observability and efficiency tuning, instructing individuals how one can monitor useful resource utilization, community throughput (EFA site visitors), and system well being. By the top of those enablement periods, prospects and supporting AWS engineers had established a shared baseline of data and a toolkit of greatest practices. Utilizing the property and information gained throughout the workshops, prospects participated in onboarding periods—structured, hands-on conferences with their Lead SAs. These periods differed from the sooner workshops by specializing in customer-specific cluster deployments tailor-made to every crew’s distinctive use case. Throughout every session, Lead SAs labored instantly with groups to deploy coaching environments, validate setup utilizing NCCL assessments, and resolve technical points in actual time.
Buyer suggestions
“To basically clear up information entry challenges, we considerably improved processing accuracy and cost-efficiency by making use of two-stage reasoning and autonomous studying with SLM and LLM for normal objects, and visible studying with VLM utilizing 100,000 artificial information samples for detailed objects. We additionally utilized Amazon EC2 P5 cases to boost analysis and growth effectivity. These formidable initiatives had been made doable because of the help of many individuals, together with AWS. We’re deeply grateful for his or her in depth help.”
– Takuma Inoue, Government Officer, CTO at AI Inside
“Future selected AWS to develop large-scale language fashions specialised for Japanese and software program growth at GENIAC. When coaching large-scale fashions utilizing a number of nodes, Future had issues about surroundings settings comparable to inter-node communication, however AWS had a variety of instruments, comparable to AWS ParallelCluster, and we obtained robust help from AWS Options Architects, which enabled us to begin large-scale coaching shortly.”
– Makoto Morishita, Chief Analysis Engineer at Future
Outcomes and looking out forward
GENIAC has demonstrated that coaching FMs at scale is basically an organizational problem, not merely a {hardware} one. By structured help, reproducible templates, and a cross-functional engagement crew (WWSO Frameworks Crew, Lead SAs, and Account Groups), even small groups can efficiently execute large workloads within the cloud. Due to this construction, 12 prospects launched over 127 P5 cases and 24 Trn1 cases throughout a number of AWS Areas, together with Asia Pacific (Tokyo), in a single day. A number of giant language fashions (LLMs) and customized fashions had been educated efficiently, together with a 32B multimodal mannequin on Trainium and a 405B tourism-focused multilingual mannequin.The technical engagement framework established by way of GENIAC Cycle 2 has offered essential insights into large-scale FM growth. Constructing on this expertise, AWS is advancing enhancements throughout a number of dimensions: engagement fashions, technical property, and implementation steering. We’re strengthening cross-functional collaboration and systematizing information sharing to ascertain a extra environment friendly help construction. Reference architectures and automatic coaching templates proceed to be enhanced, and sensible technical workshops and greatest practices are being codified primarily based on classes discovered.AWS has already begun preparations for the subsequent cycle of GENIAC. As a part of the onboarding course of, AWS hosted a complete technical occasion in Tokyo on April 3, 2025, to equip FM builders with hands-on expertise and architectural steering. The occasion, attended by over 50 individuals, showcased the dedication AWS has to supporting scalable, resilient generative AI infrastructure.
The occasion highlighted the technical engagement mannequin of AWS for GENIAC, alongside different help mechanisms, together with the LLM Growth Assist Program and Generative AI Accelerator. The day featured an intensive workshop on SageMaker HyperPod and Slurm, the place individuals gained hands-on expertise with multi-node GPU clusters, distributed PyTorch coaching, and observability instruments. Classes coated important matters, together with containerized ML, distributed coaching methods, and AWS purpose-built silicon options. Classmethod Inc. shared sensible SageMaker HyperPod insights, and AWS engineers demonstrated architectural patterns for large-scale GPU workloads. The occasion showcased AWS’s end-to-end generative AI help panorama, from infrastructure to deployment instruments, setting the stage for GENIAC Cycle 3. As AWS continues to broaden its help for FM growth, the success of GENIAC serves as a blueprint for enabling organizations to construct and scale their AI capabilities successfully.
By these initiatives, AWS will proceed to offer sturdy technical help, facilitating the graceful execution of large-scale FM coaching. We stay dedicated to contributing to the development of generative AI growth everywhere in the world by way of our technical experience.
This submit was contributed by AWS GENIAC Cycle 2 core members Masato Kobayashi, Kenta Ichiyanagi, and Satoshi Shirasawa, Accelerated Computing Specialist Mai Kiuchi, in addition to Lead SAs Daisuke Miyamoto, Yoshitaka Haribara, Kei Sasaki, Soh Ohara, and Hiroshi Tokoyo with Government Sponsorship from Toshi Yasuda. Hiroshi Hata and Tatsuya Urabe additionally offered help as core member and Lead SA throughout their time at AWS.
The authors prolong their gratitude to WWSO Frameworks members Maxime Hugues, Matthew Nightingale, Aman Shanbhag, Alex Iankoulski, Anoop Saha, Yashesh Shroff, Natarajan Chennimalai Kumar, Shubha Kumbadakone, and Sundar Ranganathan for his or her technical contributions. Pierre-Yves Aquilanti offered in-depth help throughout his time at AWS.
Concerning the authors
Keita Watanabe is a Senior Specialist Options Architect on the AWS WWSO Frameworks crew. His background is in machine studying analysis and growth. Previous to becoming a member of AWS, Keita labored within the ecommerce business as a analysis scientist growing picture retrieval programs for product search. He leads GENIAC technical engagements.
Masaru Isaka is a Principal Enterprise Growth on the AWS WWSO Frameworks crew, specializing in machine studying and generative AI options. Having engaged with GENIAC since its inception, he leads go-to-market methods for AWS’s generative AI choices.
