Supercharge your LLM efficiency with Amazon SageMaker Massive Mannequin Inference container v15

Right this moment, we’re excited to announce the launch of Amazon SageMaker Massive Mannequin Inference (LMI) container v15, powered by vLLM 0.8.4 with assist for the vLLM V1 engine. This model now helps the newest open-source fashions, akin to Meta’s Llama 4 fashions Scout and Maverick, Google’s Gemma 3, Alibaba’s Qwen, Mistral AI, DeepSeek-R, and lots of extra. Amazon SageMaker AI continues to evolve its generative AI inference capabilities to fulfill the rising calls for in efficiency and mannequin assist for basis fashions (FMs).

This launch introduces important efficiency enhancements, expanded mannequin compatibility with multimodality (that’s, the flexibility to grasp and analyze text-to-text, images-to-text, and text-to-images knowledge), and gives built-in integration with vLLM that will help you seamlessly deploy and serve massive language fashions (LLMs) with the very best efficiency at scale.

What’s new?

LMI v15 brings a number of enhancements that enhance throughput, latency, and usefulness:

1. An async mode that straight integrates with vLLM’s AsyncLLMEngine for improved request dealing with. This mode creates a extra environment friendly background loop that constantly processes incoming requests, enabling it to deal with a number of concurrent requests and stream outputs with greater throughput than the earlier Rolling-Batch implementation in v14.
2. Help for the vLLM V1 engine, which delivers as much as 111% greater throughput in comparison with the earlier V0 engine for smaller fashions at excessive concurrency. This efficiency enchancment comes from diminished CPU overhead, optimized execution paths, and extra environment friendly useful resource utilization within the V1 structure. LMI v15 helps each V1 and V0 engines, with V1 being the default. When you have a necessity to make use of V0, you should utilize the V0 engine by specifying VLLM_USE_V1=0. vLLM V1’s engine additionally comes with a core re-architecture of the serving engine with simplified scheduling, zero-overhead prefix caching, clear tensor-parallel inference, environment friendly enter preparation, and superior optimizations with torch.compile and Flash Consideration 3. For extra data, see the vLLM Weblog.
3. Expanded API schema assist with three versatile choices to permit seamless integration with functions constructed on fashionable API patterns:
   1. Message format suitable with the OpenAI Chat Completions API.
   2. OpenAI Completions format.
   3. Textual content Era Inference (TGI) schema to assist backward compatibility with older fashions.
4. Multimodal assist, with enhanced capabilities for vision-language fashions together with optimizations akin to multimodal prefix caching
5. Constructed-in assist for operate calling and power calling, enabling refined agent-based workflows.

Enhanced mannequin assist

LMI v15 helps an increasing roster of state-of-the-art fashions, together with the newest releases from main mannequin suppliers. The container presents ready-to-deploy compatibility for however not restricted to:

• Llama 4 – Llama-4-Scout-17B-16E and Llama-4-Maverick-17B-128E-Instruct
• Gemma 3 – Google’s light-weight and environment friendly fashions, identified for his or her sturdy efficiency regardless of smaller dimension
• Qwen 2.5 – Alibaba’s superior fashions together with QwQ 2.5 and Qwen2-VL with multimodal capabilities
• Mistral AI fashions – Excessive-performance fashions from Mistral AI that supply environment friendly scaling and specialised capabilities
• DeepSeek-R1/V3 – Cutting-edge reasoning fashions

Every mannequin household might be deployed utilizing the LMI v15 container by specifying the suitable mannequin ID, for instance, meta-llama/Llama-4-Scout-17B-16E, and configuration parameters as surroundings variables, with out requiring customized code or optimization work.

Benchmarks

Our benchmarks display the efficiency benefits of LMI v15’s V1 engine in comparison with earlier variations:

DeepSeek-R1 Llama 70B for numerous ranges of concurrency

Llama 3.1 8B Instruct for numerous stage of concurrency

Mistral 7B for numerous ranges of concurrency

The async engine in LMI v15 reveals energy in high-concurrency situations, the place a number of simultaneous requests profit from the optimized request dealing with. These benchmarks spotlight that the V1 engine in async mode delivers between 24% and 111% greater throughput in comparison with LMI v14 utilizing rolling batch within the fashions examined in excessive concurrency situations for batch dimension of 64 and 128. We recommend to bear in mind the next issues for optimum efficiency:

• Increased batch sizes enhance concurrency however include a pure tradeoff by way of latency
• Batch sizes of 4 and eight present one of the best latency for many use circumstances
• Batch sizes as much as 64 and 128 obtain most throughput with acceptable latency trade-offs

API codecs

LMI v15 helps three API schemas: OpenAI Chat Completions, OpenAI Completions, and TGI.

• Chat Completions – Message format is suitable with OpenAI Chat Completions API. Use this schema for instrument calling, reasoning, and multimodal use circumstances. Here’s a pattern of the invocation with the Messages API:

  physique = {
      "messages": [
          {"role": "user", "content": "Name popular places to visit in London?"}
      ],
      "temperature": 0.9,
      "max_tokens": 256,
      "stream": True,
  }

• OpenAI Completions format – The Completions API endpoint is not receiving updates:

  physique = {
   "immediate": "Title fashionable locations to go to in London?",
   "temperature": 0.9,
   "max_tokens": 256,
   "stream": True,
  } 

• TGI – Helps backward compatibility with older fashions:

  physique = {
  "inputs": "Title fashionable locations to go to in London?",
  "parameters": {
  "max_new_tokens": 256,
  "temperature": 0.9,
  },
  "stream": True,
  }

Getting began with LMI v15

Getting began with LMI v15 is seamless, and you may deploy with LMI v15 in just a few strains of code. The container is on the market by Amazon Elastic Container Registry (Amazon ECR), and deployments might be managed by SageMaker AI endpoints. To deploy fashions, it’s essential to specify the Hugging Face mannequin ID, occasion kind, and configuration choices as surroundings variables.

For optimum efficiency, we suggest the next situations:

• Llama 4 Scout: ml.p5.48xlarge
• DeepSeek R1/V3: ml.p5e.48xlarge
• Qwen 2.5 VL-32B: ml.g5.12xlarge
• Qwen QwQ 32B: ml.g5.12xlarge
• Mistral Massive: ml.g6e.48xlarge
• Gemma3-27B: ml.g5.12xlarge
• Llama 3.3-70B: ml.p4d.24xlarge

To deploy with LMI v15, comply with these steps:

1. Clone the pocket book to your Amazon SageMaker Studio pocket book or to Visible Studio Code (VS Code). You possibly can then run the pocket book to do the preliminary setup and deploy the mannequin from the Hugging Face repository to the SageMaker AI endpoint. We stroll by the important thing blocks right here.
2. LMI v15 maintains the identical configuration sample as earlier variations, utilizing surroundings variables within the kind OPTION_. This constant method makes it simple for customers conversant in earlier LMI variations emigrate to v15.

   vllm_config = {
       "HF_MODEL_ID": "meta-llama/Llama-4-Scout-17B-16E",
       "HF_TOKEN": "entertoken",
       "OPTION_MAX_MODEL_LEN": "250000",
       "OPTION_MAX_ROLLING_BATCH_SIZE": "8",
       "OPTION_MODEL_LOADING_TIMEOUT": "1500",
       "SERVING_FAIL_FAST": "true",
       "OPTION_ROLLING_BATCH": "disable",
       "OPTION_ASYNC_MODE": "true",
       "OPTION_ENTRYPOINT": "djl_python.lmi_vllm.vllm_async_service"
   }

   • HF_MODEL_ID units the mannequin id from Hugging Face. It’s also possible to obtain mannequin from Amazon Easy Storage Service (Amazon S3).
   • HF_TOKEN units the token to obtain the mannequin. That is required for gated fashions like Llama-4
   • OPTION_MAX_MODEL_LEN. That is the max mannequin context size.
   • OPTION_MAX_ROLLING_BATCH_SIZE units the batch dimension for the mannequin.
   • OPTION_MODEL_LOADING_TIMEOUT units the timeout worth for SageMaker to load the mannequin and run well being checks.
   • SERVING_FAIL_FAST=true. We suggest setting this flag as a result of it permits SageMaker to gracefully restart the container when an unrecoverable engine error happens.
   • OPTION_ROLLING_BATCH= disable disables the rolling batch implementation of LMI, which was the default providing in LMI V14. We suggest utilizing async as an alternative as this newest implementation and gives higher efficiency
   • OPTION_ASYNC_MODE=true permits async mode.
   • OPTION_ENTRYPOINT gives the entrypoint for vLLM’s async integrations
3. Set the newest container (on this instance we used 0.33.0-lmi15.0.0-cu128), AWS Area (us-east-1), and create a mannequin artifact with all of the configurations. To evaluation the newest accessible container model, see Out there Deep Studying Containers Photos.
4. Deploy the mannequin to the endpoint utilizing mannequin.deploy().

   CONTAINER_VERSION = '0.33.0-lmi15.0.0-cu128'
   REGION = 'us-east-1'
   # Assemble container URI
   container_uri = f'763104351884.dkr.ecr.{REGION}.amazonaws.com/djl-inference:{CONTAINER_VERSION}'
   
   # Choose occasion kind
   instance_type = "ml.p5.48xlarge"
   
   mannequin = Mannequin(image_uri=container_uri,
                 position=position,
                 env=vllm_config)
   endpoint_name = sagemaker.utils.name_from_base("Llama-4")
   
   print(endpoint_name)
   mannequin.deploy(
       initial_instance_count=1,
       instance_type=instance_type,
       endpoint_name=endpoint_name,
       container_startup_health_check_timeout = 1800
   )

5. Invoke the mannequin, SageMaker inference gives two APIs to invoke the model- InvokeEndpoint and InvokeEndpointWithResponseStream. You possibly can select both possibility primarily based in your wants.

   # Create SageMaker Runtime shopper
   smr_client = boto3.shopper('sagemaker-runtime')
   ##Add your endpoint right here 
   endpoint_name=""
   
   # Invoke with messages format
   physique = {
   "messages": [
   {"role": "user", "content": "Name popular places to visit in London?"}
   ],
   "temperature": 0.9,
   "max_tokens": 256,
   "stream": True,
   }
   
   # Invoke with endpoint streaming
   resp = smr_client.invoke_endpoint_with_response_stream(
   EndpointName=endpoint_name,
   Physique=json.dumps(physique),
   ContentType="utility/json",
   )

To run multi-modal inference with Llama-4 Scout, see the pocket book for the total code pattern to run inference requests with pictures.

Conclusion

Amazon SageMaker LMI container v15 represents a major step ahead in massive mannequin inference capabilities. With the brand new vLLM V1 engine, async working mode, expanded mannequin assist, and optimized efficiency, you possibly can deploy cutting-edge LLMs with better efficiency and suppleness. The container’s configurable choices provide the flexibility to fine-tune deployments on your particular wants, whether or not optimizing for latency, throughput, or value.

We encourage you to discover this launch for deploying your generative AI fashions.

Take a look at the offered instance notebooks to start out deploying fashions with LMI v15.

In regards to the authors

Vivek Gangasani is a Lead Specialist Options Architect for Inference at AWS. He helps rising generative AI firms construct modern options utilizing AWS companies and accelerated compute. Presently, he’s centered on growing methods for fine-tuning and optimizing the inference efficiency of huge language fashions. In his free time, Vivek enjoys mountain climbing, watching films, and attempting completely different cuisines.

Siddharth Venkatesan is a Software program Engineer in AWS Deep Studying. He at present focusses on constructing options for big mannequin inference. Previous to AWS he labored within the Amazon Grocery org constructing new fee options for patrons world-wide. Outdoors of labor, he enjoys snowboarding, the outside, and watching sports activities.

Felipe Lopez is a Senior AI/ML Specialist Options Architect at AWS. Previous to becoming a member of AWS, Felipe labored with GE Digital and SLB, the place he centered on modeling and optimization merchandise for industrial functions.

Banu Nagasundaram leads product, engineering, and strategic partnerships for Amazon SageMaker JumpStart, the SageMaker machine studying and generative AI hub. She is captivated with constructing options that assist clients speed up their AI journey and unlock enterprise worth.

Dmitry Soldatkin is a Senior AI/ML Options Architect at Amazon Internet Providers (AWS), serving to clients design and construct AI/ML options. Dmitry’s work covers a variety of ML use circumstances, with a major curiosity in Generative AI, deep studying, and scaling ML throughout the enterprise. He has helped firms in lots of industries, together with insurance coverage, monetary companies, utilities, and telecommunications. You possibly can join with Dmitry on LinkedIn.