Prevent CloudFormation failures when updating SageMaker AI endpoints

Removing and reattaching Application Auto Scaling during deployments removes a common source of failed SageMaker endpoint updates, saving developer time and deployment risk.

What actually changed

AWS found that updating a SageMaker AI real-time endpoint configuration (for example changing the instance type, instance count or environment variables) can fail when Application Auto Scaling is registered through CloudFormation. SageMaker requires the scalable target be deregistered before the endpoint update proceeds; if it is still registered, CloudFormation update attempts are rejected. The AWS Machine Learning Blog demonstrates a pattern: provide a reusable AWS CDK construct that uses CloudFormation custom resources to orchestrate the deregister and subsequent register of auto scaling automatically during endpoint updates. Using that construct allows endpoint configuration and auto scaling parameter updates in a single deployment rather than requiring separate manual steps.

"deregister and register auto scaling"

Who it affects

• Small engineering or DevOps teams that deploy SageMaker AI real-time endpoints via AWS CloudFormation or the AWS CDK.
• Teams that attach Application Auto Scaling (scalable targets and scaling policies) to SageMaker endpoints and make frequent configuration changes.
• Any pipeline where failed CloudFormation updates force manual intervention, rollback work or interrupted deployments.

If you run SageMaker endpoints but don't use CloudFormation/CDK to manage the auto scaling resources, this specific failure mode is less likely to affect you.

What it costs or what it replaces

• The announcement does not state pricing for the CDK construct or any managed artefact. There is no price information in the blog summary.
• Practically, the approach replaces the manual process of deregistering scalable targets, updating the endpoint, and re-registering auto scaling. That reduces hands-on time during deployments and the risk of CloudFormation update failures.
• It also replaces ad-hoc scripts and manual change windows with an automated step inside your CDK stack, so the operational cost shifts from manual labour to a one-time integration task inside your infrastructure-as-code.

What we don't know

• Where the reusable CDK construct is published (repository or CDK construct library) and which languages (TypeScript, Python, Java, etc.) are supported.
• Exact implementation details: the custom resource code, IAM permissions required, and how it handles errors and retries.
• Whether the construct preserves existing scaling policies and alarms automatically when it re-registers the scalable target.
• Behaviour in complex stacks with multiple interdependent CloudFormation resources or cross-stack references.
• Any limits, performance implications or rollback behaviour when the custom resource fails during an update.

What to do next

1. Inventory: In your staging account, list SageMaker real-time endpoints managed by CloudFormation/CDK and identify which have Application Auto Scaling scalable targets attached (check the Application Auto Scaling console and your CDK/CloudFormation templates).
2. Prototype: Follow the blog's pattern by adding a reusable CDK construct that uses a CloudFormation custom resource to deregister the scalable target before endpoint updates and re-register it afterwards. Deploy the change to a non-production environment and monitor CloudFormation events and Application Auto Scaling registration.
3. Test and observe: Make a controlled change that previously caused failures (instance type, instance count or environment variable update). Confirm the stack update completes without manual deregistration, validate scaling policies remain as expected, and document any IAM or rollback adjustments required for your organisation.