AWS Splits One GPU Into Slices: 18-34% Cheaper AI Hosting, With a Catch
In short
- Amazon SageMaker can now split one NVIDIA GPU into hardware-isolated slices, each running its own model.
- AWS measured 18-34% lower cost per million output tokens on a half H100 versus a whole A100, but only above a certain traffic level.
- No extra fee. You pay the normal price of the instance and use more of it.
Paying for a whole GPU to run a small model is like renting a truck to deliver one pizza. AWS just added a way to stop doing that.
What happened
On November 6, the AWS Machine Learning Blog announced GPU partitioning for SageMaker Inference, built on NVIDIA Multi-Instance GPU (MIG). One GPU becomes several isolated partitions. Each partition can host a different model, or one model can sit on a fraction of a GPU.
It works on endpoints that use inference components. You turn it on with two optional API fields: AcceleratorPartitionLayouts on the production variant and AcceleratorPartitionType on each component. No new API operations, no new permissions.
Supported hardware: P4d (A100 40 GB), P5 (H100 80 GB), G7 (RTX PRO 4500 Blackwell, 32 GB) and G7e (RTX PRO 6000 Blackwell Server, 96 GB). Regions include Frankfurt and Ireland, plus US East, US West, Singapore and Tokyo.
The headline number comes with a catch. AWS compared one H100 half-GPU partition against one whole A100-40GB, both serving Llama-3.1-8B-Instruct. The partition was 18-34% cheaper per million output tokens. But only above a concurrency crossover: 64 simultaneous requests for 100-token prompts, just 4 for 8,000-token prompts. Below that, the whole A100 is cheaper. AWS says it is a case-by-case comparison, not a general verdict.
How to use it
- Several small models, one box. A classifier, an embedder and a chat model each get their own slice instead of three instances.
- Busy endpoints only. If your traffic is spiky and low, partitioning will not help. Measure your concurrency first.
- Check the P4d detail. The mig-1g.10gb profile needs InferenceAmiVersion al2023-ami-sagemaker-inference-gpu-4-1.
How much does it cost? Nothing extra. AWS bills the full instance at its normal hourly rate. A p4d.24xlarge in Oregon was $25.25 an hour in AWS’s analysis.
Do the math: imagine you spend $10,000 a month on inference for a small model under heavy traffic. A saving of 18-34% is $1,800 to $3,400 a month. At roughly €0.86 per dollar (my assumption), that is about €1,550 to €2,900 a month, or €18,600 to €35,000 a year. Your saving can be zero if traffic is below the crossover.
What it means
The small-model era needs small-model economics. Teams that run many 8B-class models have been burning GPU hours on idle silicon. Partitioning pushes utilisation up without touching the model.
Who loses? Anyone selling thin wrappers around per-token pricing for small models. Cheaper hosting squeezes their margin. And note the test: one model, one comparison, run by the seller. Run your own load test.
Ivan’s Take
Good for teams already on SageMaker with busy small models. Skip it if you are small and your traffic is light, the crossover will eat the saving. I would run one load test at my real concurrency before moving anything. That test costs a few dollars. A wrong guess costs thousands.
Source: AWS
If it doesn’t save you time or money, you won’t see it here
- Facts, not hypeEvery price and number is checked against the official source
- Always the real priceYou see what it costs before you click
- I say when it’s weakEven when it’s OpenAI or Google
