Operating an endpoint is different from operating it efficiently. Unlike training costs, inference costs persist for the entire model lifespan, so optimization directly impacts total cost. SageMaker provides multiple tools to make models faster (lower latency), cheaper (lower cost), and smaller (lighter resource footprint). Today covers SageMaker Neo, Elastic Inference, AWS Inferentia, and throughput strategies like inference pipelines and batching.
Neo compiles and optimizes trained models for specific hardware (instances or edge devices)