AWS adds GPU partitioning to SageMaker AI real-time endpoints
AWS announced on October 30 that Amazon SageMaker AI real-time inference endpoints support GPU partitioning. The feature uses NVIDIA Multi-Instance GPU technology to divide a GPU into as many as seven isolated partitions, each with its own memory and compute resources.
The announcement targets small and mid-size models, including fine-tuned models, embedding models, rerankers and classifiers. AWS says these workloads previously needed a whole accelerator for each model, leaving some GPU capacity unused.
With the new option, teams choose how to divide the GPUs behind an endpoint and place each model on a partition sized for that workload. AWS describes the separation as hardware-enforced isolation intended to keep one busy model from taking memory or throughput from neighboring models.
Partitioning is configured when creating an endpoint that hosts inference components. Models continue to be invoked by name, so AWS says the application code used to call them does not need to change. This does not remove the need to size and configure the deployment itself.
AWS lists support on P4d, P5, G7 and G7e instances in seven regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt) and Europe (Ireland). The announcement is not a statement of availability on every instance type or in every AWS region.
The practical change is the ability to share a supported accelerator across appropriately sized models while keeping separate hardware partitions. AWS does not publish a percentage cost saving or a latency guarantee in this announcement. Any savings depend on the workload and deployment choices rather than a guaranteed discount stated in the release.