Construct AI workflows on Amazon EKS with Union.ai and Flyte

As synthetic intelligence and machine studying (AI/ML) workflows develop in scale and complexity, it turns into more durable for practitioners to prepare and deploy their fashions. AI initiatives typically battle to maneuver from pilot to manufacturing. AI initiatives typically fail not as a result of fashions are unhealthy, however as a result of infrastructure and processes are fragmented and brittle, and the unique pilot code base is usually compelled to bloat by these further necessities. This makes it troublesome for knowledge scientists and engineers to shortly transfer from laptop computer to cluster (native improvement to manufacturing deployment) and reproduce the precise outcomes they’d seen throughout the pilot.

On this publish, we clarify how you should utilize the Flyte Python SDK to orchestrate and scale AI/ML workflows. We discover how the Union.ai 2.0 system permits deployment of Flyte on Amazon Elastic Kubernetes Service (Amazon EKS), integrating seamlessly with AWS providers like Amazon Easy Storage Service (Amazon S3), Amazon Aurora, AWS Id and Entry Administration (IAM), and Amazon CloudWatch. We discover the answer by way of an AI workflow instance, utilizing the brand new Amazon S3 Vectors service.

Frequent challenges operating AI/ML workflows on Kubernetes

AI/ML workflows operating on Kubernetes current a number of orchestration challenges:

• Infrastructure complexity – Provisioning the fitting compute sources (CPUs, GPUs, reminiscence) dynamically throughout Kubernetes clusters
• Experiment-to-production hole – Transferring from experimentation to manufacturing typically requires rebuilding pipelines in several environments
• Reproducibility – Monitoring knowledge lineage, mannequin variations, and experiment parameters to facilitate dependable outcomes
• Value administration – Effectively using spot situations, automated scaling, and avoiding over-provisioning
• Reliability – Dealing with failures gracefully with automated retries, checkpointing, and restoration mechanisms

Goal-built AI/ML tooling is important for orchestrating complicated workflows, providing specialised capabilities like clever caching, automated versioning, and dynamic useful resource allocation that streamline improvement and deployment cycles.

Why Flyte/Union for Amazon EKS

The Flyte on Amazon EKS Python workflows scale from laptop-to-cluster with dynamic execution, reproducibility, and compute-aware orchestration. These workflows, together with Union.ai’s managed deployment, facilitate seamless, crash-proof operations that absolutely make the most of Amazon EKS with out the infrastructure overhead. Flyte transforms how one can orchestrate AI/ML workloads on Amazon EKS, making workflows easy to construct. Some key elements embody:

• Pure Python workflows – Write orchestration logic in Python with 66% much less code than conventional orchestrators, assuaging the necessity to be taught domain-specific languages and eradicating boundaries for ML engineers and AI builders migrating present code
• Dynamic execution – Make real-time selections at runtime with versatile branching, loops, and conditional logic, which is important for agentic AI programs
• Reproducibility by default – Each execution is versioned, cached, and tracked with full knowledge lineage
• Compute-aware orchestration – Dynamically provision the fitting compute sources for every job, from CPUs for knowledge processing to GPUs for mannequin coaching
• Robustness – Pipelines can shortly get well from failures, isolate errors, and handle checkpoints with out guide intervention

Union.ai 2.0 is constructed on Flyte, the open supply, Kubernetes-based workflow orchestration system initially developed at Lyft to energy mission-critical ML programs like ETA prediction, pricing, and mapping. After Flyte was open sourced in 2020 and have become a Linux Basis AI & Information mission, the core engineering group based Union.ai 2.0 to ship an enterprise-grade service purposed-built for groups operating AI/ML workloads on Amazon EKS. Union.ai 2.0 reduces the complexity of managing Kubernetes infrastructure by way of managed operations, a multi-cloud management airplane, and abstracted infrastructure administration, whereas offering ML-based capabilities that assist knowledge scientists and engineers concentrate on constructing fashions with enhanced scale, velocity, safety, and reliability.

Further advantages of utilizing Union.ai 2.0 embody:

• Enhanced scalability – Workflows reply at runtime with versatile branching, job fanout, and real-time infrastructure scaling.
• Crash-proof reliability – Computerized retries, checkpointing, and failure restoration enable workflows to remain resilient with out guide intervention.
• Agentic AI runtime – Union.ai is designed for long-lived agentic AI programs, supporting stateful brokers and actually sturdy orchestration.
• Compliance – For regulated industries, built-in lineage, auditability, and safe execution (SOC2, RBAC, SSO) are vital. Orchestration on Amazon EKS and Union.ai helps facilitate compliance.
• Useful resource consciousness – It gives first-class assist for compute provisioning, spot situations, and automated scaling.

The advantages of Flyte and Union.ai 2.0 elevate trendy orchestration to a first-class requirement: dynamic execution, fault tolerance, and useful resource consciousness at the moment are built-in, offering a extra developer-friendly expertise in comparison with 1.0.

Amazon EKS offers your compute, storage, and networking spine. Flyte (the open supply mission) handles workflow orchestration. Union.ai extends Flyte with infrastructure-aware orchestration, enterprise-grade safety, and turnkey scalability, providing you with production-ready Flyte with out the DIY setup. Each Flyte and Union.ai 2.0 run on Amazon EKS, however serve completely different wants, as detailed within the following desk.

Enterprises like Woven Toyota, Lockheed Martin, Spotify, and Artera orchestrate thousands and thousands of {dollars} of compute yearly with Flyte and Union, accelerating experimentation by 25 occasions quicker and chopping iteration cycles by 96%.

Each choices (open supply Flyte and Union.ai 2.0) combine with the open supply neighborhood, facilitating fast function rollout and steady enchancment.

Resolution overview

Though open supply Flyte offers highly effective orchestration capabilities, Union.ai 2.0 delivers the identical core expertise with enterprise-grade administration, eradicating the operational overhead so your group can concentrate on constructing AI functions as an alternative of managing infrastructure. That is achieved by way of a hybrid structure that mixes managed simplicity with full knowledge management. The Regional management airplane handles workflow metadata and coordination, whereas the Union Operator deploys straight into your EKS clusters—retaining your knowledge, code, and secrets and techniques totally inside your AWS perimeter.

The next determine illustrates the operational stream between Union’s management airplane and your knowledge airplane. The Union-managed management airplane (left) orchestrates workflows by way of Elastic Load Balancing (ELB), storing job knowledge in Amazon S3 and execution metadata in Aurora. Inside your Amazon EKS atmosphere (proper), the info airplane executes workflows that pull buyer code out of your container registry, entry secrets and techniques from AWS Secrets and techniques Supervisor, and browse/write knowledge to your S3 buckets—with the execution logs flowing to each CloudWatch and the Union management airplane for observability.

Union.ai 2.0’s AWS integration structure is constructed on six key service parts that present end-to-end workflow administration:

• Management airplane and knowledge airplane – The management airplane operates inside the Union.ai AWS account and serves because the central administration interface, offering customers with authentication and authorization capabilities, commentary and monitoring features, and system administration instruments. It additionally orchestrates execution placement on knowledge airplane clusters and handles cluster management and administration operations. Union.ai 2.0 maintains one management airplane per AWS Area, managing the Regional knowledge planes. Accessible Areas for knowledge airplane deployment embody us-west, us-east, eu-west, and eu-central, with ongoing enlargement to further Areas.
• Information airplane object retailer – This element shops knowledge comprising information, directories, knowledge frames, fashions, and Python-pickled varieties, that are handed as references and browse by the management airplane.
• Container registry – This element accommodates registry knowledge that embody names of workflows, duties, launch plans, and artifacts; enter and output varieties for workflows and duties; execution standing, begin time, finish time, and period of workflows and duties; model info for workflows, duties, launch plans, and artifacts; and artifact definitions. With the Union.ai 2.0 structure, you may retain full possession of your knowledge and compute sources whereas it manages the infrastructure operations. The Union.ai 2.0 operator resides within the knowledge airplane and handles administration duties with least privilege permissions. It permits cluster lifecycle operations and offers assist engineers with system-level log entry and alter implementation capabilities—with out exposing secrets and techniques or knowledge. Safety is additional strengthened by way of unidirectional communication: the info airplane operator initiates the connections to the management airplane, not the reverse.
• Logging and monitoring – CloudWatch offers centralized logging and monitoring by way of deep integration with Flyte. The system robotically builds logging hyperlinks for every execution and shows them within the console, with hyperlinks pointing on to the AWS Administration Console and the particular log stream for that execution—a function that considerably accelerates troubleshooting throughout failures.
• Safety – Safety is dealt with by way of IAM roles for service accounts (IRSA), which maps the identification between Kubernetes sources and the AWS providers they rely upon. These configurations allow safer, fine-grained entry management for backend providers, and Union.ai 2.0 provides enterprise role-based entry management (RBAC) for consumer entry management on prime of those AWS security measures.
• Storage layer – Amazon S3 serves because the sturdy storage layer for workflows and knowledge. Whenever you register a workflow with Flyte, your code is compiled right into a language-independent illustration that captures the workflow definition, enter, and output varieties. This illustration is packaged and saved in Amazon S3, the place FlytePropeller—Flyte’s execution engine—retrieves it to instruct the respective compute framework (reminiscent of Kubernetes or Spark) to run workflows and report standing. Uncooked enter knowledge used to coach and validate fashions can be saved in Amazon S3. Union.ai 2.0 now features a new integration with Amazon S3 Vectors, enabling vector storage for Retrieval Augmented Technology (RAG), semantic search, and agentic AI workflows.

With this strong infrastructure in place, Union.ai 2.0 on Amazon EKS excels at orchestrating a variety of AI/ML workloads. It handles large-scale mannequin coaching by orchestrating distributed coaching pipelines throughout GPU clusters with automated useful resource provisioning and spot occasion assist. For knowledge processing, it might course of petabyte-scale datasets with dynamic parallelism and environment friendly job fanout, scaling to 100,000 job fanouts with 50,000 concurrent actions in Union.ai 2.0. Through the use of Union.ai 2.0 and Flyte on Amazon EKS, you may construct and deploy agentic AI programs—long-running, stateful AI brokers that make autonomous selections at runtime. For manufacturing deployments, it helps real-time inference with low-latency mannequin serving, utilizing reusable containers for sub-100 millisecond job startup occasions. All through the whole course of, Union.ai 2.0 offers complete MLOps and mannequin lifecycle administration, automating every little thing from experimentation to manufacturing deployment with built-in versioning and rollback capabilities.

These capabilities are exemplified in specialised implementations like distributed coaching on AWS Trainium situations, the place Flyte orchestrates large-scale coaching workloads on Amazon EKS.

Deployment choices for Union.ai 2.0 on Amazon EKS

Union.ai 2.0 and Flyte provide three versatile deployment fashions for Amazon EKS, every balancing managed comfort with operational management. Choose the strategy that most closely fits your group’s experience, compliance necessities, and improvement velocity:

• Union BYOC (absolutely managed) – The quickest path to manufacturing. Union.ai 2.0 manages the infrastructure, upgrades, and scaling whereas your workloads run in your AWS account. This feature is right for groups that need to focus totally on AI improvement fairly than infrastructure operations.
• Union Self Managed – You possibly can deploy Union.ai 2.0’s managed management airplane whereas sustaining management of your knowledge and compute sources in your AWS account. This feature combines the advantages of managed providers with knowledge sovereignty and governance necessities.
• Flyte OSS on Amazon EKS – You possibly can deploy and function open supply Flyte straight in your EKS cluster utilizing the AWS Cloud Improvement Package (AWS CDK). This feature offers most management and is right for groups with sturdy Kubernetes experience who need to customise their deployment. (edited) 

The Amazon EKS Blueprints for AWS CDK Union add-on helps AWS prospects deploy, scale, and optimize AI/ML workloads utilizing Union on Amazon EKS. It offers modular infrastructure as code (IaC) AWS CDK templates and curated deployment blueprints for operating scalable AI workloads, together with:

• Mannequin coaching and fine-tuning pipelines
• Giant language mannequin (LLM) inference and serving
• Multi-model deployment and administration
• Agentic AI pipeline orchestration

Union.ai 2.0 and Flyte present IaC templates for deploying on Amazon EKS:

• Terraform modules – Preconfigured modules for deploying Flyte on Amazon EKS with greatest practices for networking, safety, and observability
• AWS CDK assist – AWS CDK constructs for integrating Union into present AWS infrastructure
• GitOps workflows – Assist for Flux and ArgoCD for declarative infrastructure administration

The Union add-on is accessible by weblog publication, and the Flyte add-on is coming—preserve watching the GitHub repo.

These templates automate the provisioning of EKS clusters, node teams (together with GPU situations), IAM roles, S3 buckets, Aurora databases, and the required Flyte parts.

Stipulations

To begin utilizing this resolution, you have to have the next stipulations:

• An AWS account with applicable permissions.
• Amazon EKS model on commonplace assist.
• Required IAM roles. Utilizing IAM roles for service accounts, Flyte can map identification between the Kubernetes sources and AWS providers it relies on. These configurations are for the backend and don’t intrude with user-control airplane communication

How Union.ai 2.0 helps Amazon S3 Vectors

As AI functions more and more depend on vector embeddings for semantic search and RAG, Union.ai 2.0 empowers groups with Amazon S3 Vectors integration, simplifying vector knowledge administration at scale. Constructed into Flyte 2.0, this function is accessible right now. Amazon S3 Vectors delivers purpose-built, cost-optimized vector storage for semantic search and AI functions. With Amazon S3 degree elasticity and sturdiness for storing vector datasets with subsecond question efficiency, Amazon S3 Vectors is right for functions that must construct and develop vector indexes at scale. Union.ai 2.0 offers assist for Amazon S3 Vectors for RAG, semantic search, and multi-agent programs. Should you’re utilizing Union.ai 2.0 right now with Amazon S3 as your object retailer, you can begin utilizing Amazon S3 Vectors instantly with minimal configuration adjustments.

To set it up, use Boto’s devoted APIs to retailer and question vectors. Your Amazon S3 IAM roles are already in place. Simply replace the permissions.

By combining Flyte 2.0’s orchestration with Amazon S3 Vector assist, multi-agent buying and selling simulations can scale to lots of of brokers that be taught from historic knowledge, share trade insights, and execute coordinated methods in actual time. These architectural benefits assist subtle AI functions like multi-agent programs that require each semantic reminiscence and real-time coordination.

To be taught extra, seek advice from the instance use case of a multi-agent buying and selling simulation utilizing Flyte 2.0 with Amazon S3 Vectors. On this instance, you’ll be taught to construct a buying and selling simulation that includes a number of brokers that characterize group members in a agency, illustrating their interactions, strategic planning, and collaborative buying and selling actions

Contemplate a multi-agent buying and selling simulation the place AI brokers work together, take a look at methods, and constantly be taught from their experiences. For life like agent habits, every agent should retain context from earlier interactions, primarily constructing a reminiscence of semantic artifacts that inform future selections. The method consists of the next steps:

1. After every simulation spherical, embed the agent’s learnings into vector representations utilizing embedding fashions.
2. Retailer embeddings in Amazon S3 utilizing Amazon S3 Vectors with applicable metadata and tags.
3. Throughout subsequent executions, retrieve related recollections utilizing semantic search to floor agent selections in previous expertise.

With Flyte 2.0, your brokers already run in an orchestration-aware atmosphere. Amazon S3 turns into your vector retailer. It’s cheap, quick, and absolutely built-in, assuaging the necessity for separate vector databases. For the steps and related code to implement the multi-agent buying and selling simulation, seek advice from the GitHub repo.

In abstract, this structure helps ship measurable benefits for manufacturing AI programs:

• Decreased operational complexity – Consolidate your AI/ML orchestration and vector storage on a single atmosphere, assuaging the necessity to provision, preserve, and safe separate vector database infrastructure
• Important value financial savings – Amazon S3 Vectors delivers considerably decrease storage prices in comparison with purpose-built vector databases, whereas offering subsecond similarity search efficiency at scale
• Zero-friction AWS integration – Use your present Amazon S3 infrastructure, IRSA configuration, and digital personal cloud (VPC) networking—no further authentication layers or community configurations are required
• Battle-tested scalability – Construct on the 99.999999999% sturdiness and elastic scalability of Amazon S3 to assist vector datasets from gigabytes to petabytes with out re-architecture

Buyer success: Woven by Toyota

Toyota’s autonomous driving arm, Woven by Toyota, confronted challenges orchestrating complicated AI workloads for his or her autonomous driving expertise, requiring petabyte-scale knowledge processing and GPU-intensive coaching pipelines. After outgrowing their open supply Flyte implementation, they migrated to Union.ai’s managed service on AWS in 2023. The affect was transformative: over 20 occasions quicker ML iteration cycles, thousands and thousands of {dollars} in annual value financial savings by way of spot occasion optimization, and 1000’s of parallel staff enabling large scale.

“Union.ai’s wealth of experience has enabled us to focus our efforts on key ADAS-related functionalities, transfer quick, and depend on Union.ai to ship knowledge at scale,”

– Alborz Alavian, Senior Engineering Supervisor at Woven by Toyota.

Learn the complete case examine about Woven by Toyota’s migration to Union.ai.

Conclusion

Union.ai and Flyte present the inspiration for dependable, scalable AI on Amazon EKS on your AI/ML workflows, reminiscent of constructing autonomous programs, coaching LLMs, or orchestrating complicated knowledge pipelines.To get began, select your path:

In regards to the authors

ND Ngoka is Senior Options Architect at AWS with specialised concentrate on AI/ML and storage applied sciences. Guides prospects by way of complicated architectural selections, enabling them to construct resilient, scalable options that drive enterprise outcomes.

Samhita Alla is a Senior Options Engineer for Partnerships at Union.ai, the place she leads the technical execution of strategic integrations throughout the AI stack, from distributed coaching and experiment monitoring to knowledge platform integrations. She works intently with companions and cross-functional groups to judge feasibility, construct production-ready options, and ship technical content material that drives real-world adoption.

Kristy Prepare dinner is Head of Partnerships at Union.ai, the place she builds strategic alliances throughout the AI/ML ecosystem centered on sustained development. Having cast impactful partnerships at Meta, Yahoo, and Neustar she brings deep experience in operationalizing AI options at scale.

Jim Fratantoni is a GenAI Account Supervisor at AWS, centered on serving to AI startups scale and co-sell with AWS. He’s obsessed with working with founders to collectively go to market and drive enterprise buyer success.

Theo Rashid is an Utilized Scientist at Amazon constructing probabilistic machine studying and forecasting fashions. He’s an energetic open supply contributor, and is obsessed with open supply tooling throughout the machine studying stack, from probabilistic programming libraries to workflow orchestration. He holds a PhD in Epidemiology and Biostatistics from Imperial Faculty London.

Alex Fabisiak is a Senior Utilized Scientist at Amazon engaged on utilized forecasting and provide chain issues. He makes a speciality of probabilistic and causal modeling as they relate to optimum coverage selections. He holds a PhD in Finance from UCLA.