Scale Amazon ECS services based on an Amazon SQS queue - Amazon Elastic Container Service

View a markdown version of this page

Scale Amazon ECS services based on an Amazon SQS queue - Amazon Elastic Container Service
Use the backlog per task metricUse metric math to compute the metricUse step scaling for more controlProtect tasks that are processing messagesConsiderationsConfigure queue-based scaling

Scale Amazon ECS services based on an Amazon SQS queue

You can scale your Amazon ECS service in response to the load in an Amazon Simple Queue Service (Amazon SQS) queue. This is useful for asynchronous applications in which the tasks in your service act as workers that consume and process messages from a queue. As the rate of incoming messages changes, Amazon ECS Service Auto Scaling adds tasks when the queue builds up and removes tasks when it drains. This keeps latency for messages waiting in the queue at an acceptable level without over-provisioning.

Use the backlog per task metric

Amazon SQS publishes the depth of a queue to CloudWatch as the ApproximateNumberOfMessagesVisible metric, which reports the number of messages available for retrieval. Queue depth indicates that there is work to be done, but it is not a reliable metric to scale on directly. The number of messages in the queue does not change proportionally to the number of tasks that process it. The number of tasks that you need also depends on other factors, such as how long a task takes to process a message and the acceptable amount of latency (queue delay). A target tracking policy that tracks queue depth interprets a large queue as demand for proportionally more tasks. This can cause your service to scale out far beyond what the workload requires and increase cost.

Instead, scale on the backlog per task: the queue depth divided by the number of running tasks. You calculate this value with a CloudWatch metric math expression that divides the queue depth by the running task count. Backlog per task normalizes queue depth by the running task count. This makes it proportional to the load on each task, so it is suitable for scaling.

Set your target value to the acceptable backlog per task. To calculate it, divide the latency that your application can accept by the average time that a task takes to process a message.

For example, if the acceptable latency is 10 seconds and a task takes 0.1 seconds to process a message, the acceptable backlog per task is 10 divided by 0.1, which equals 100 messages per task. This is the target value for your policy. If the queue currently holds 1,500 messages and 10 tasks are running, the backlog per task is 150 (1,500 divided by 10), which exceeds the target. Your service scales out by five tasks to bring the backlog per task back in proportion to the target value.

Use metric math to compute the metric

Application Auto Scaling target tracking policies support CloudWatch metric math. Metric math lets you query multiple CloudWatch metrics and use math expressions to create a new time series based on those metrics. You use a metric math expression to compute the backlog per task directly in the scaling policy. Application Auto Scaling evaluates the expression and manages the CloudWatch alarms that start scaling activities. For more information, see Use metric math in the CloudWatch User Guide.

To compute the backlog per task, you use the following two CloudWatch metrics:

  • ApproximateNumberOfMessagesVisible (the Sum statistic) from the AWS/SQS namespace, for the number of messages waiting in the queue.

  • RunningTaskCount (the Average statistic) from the ECS/ContainerInsights namespace, for the number of running tasks in your service. To use this metric, you must turn on CloudWatch Container Insights for your cluster. For more information, see Monitor Amazon ECS containers using Container Insights with enhanced observability.

In the following example, m1 is the queue depth and m2 is the running task count. The metric math expression m1 / m2 produces the backlog per task. Application Auto Scaling adjusts the number of tasks in your service to keep this value at your target value.

Use step scaling for more control

We recommend target tracking for most queue-based workloads because it is the simplest approach. You set a single target value, and Amazon ECS Service Auto Scaling creates and manages the CloudWatch alarms and the scaling adjustments for you.

Use step scaling when you want full control over the scaling behavior. With step scaling, you create your own CloudWatch alarms and define separate scale-out and scale-in policies, each with its own step adjustments. You put the backlog per task metric math expression in the alarms, and set each alarm's action to the corresponding policy. This gives you control over the following:

  • The size of each scaling step – You can define step adjustments that add more tasks as the backlog grows, so that a small breach adds a few tasks and a large breach adds many. You can scale by a percentage of the running task count or by a fixed number of tasks.

  • How quickly scaling responds – You control responsiveness through the metric publishing frequency and the alarm evaluation settings. For example, you can publish a high-resolution queue metric every 10 seconds and configure a scale-out alarm to breach on a single data point, so that the service reacts to a growing backlog faster than the default one-minute resolution allows.

  • Different scale-out and scale-in behavior – Because scale-out and scale-in are separate policies and alarms, you can react quickly to a rising backlog while scaling in conservatively. For example, you can require the backlog to stay low for several minutes before removing tasks.

For more information, see Use predefined increments based on CloudWatch alarms to scale Amazon ECS services.

Protect tasks that are processing messages

When your service scales in, Amazon ECS Service Auto Scaling can stop a task that is still processing a message. For long-running or non-idempotent work, this can cause the message to be reprocessed or its progress to be lost. To prevent this, use task scale-in protection to mark a task as protected while it is doing active work.

For queue workers, we recommend that the task protect itself through the Amazon ECS container agent endpoint. When the task starts processing a message, it sets the ProtectionEnabled attribute to true so that Amazon ECS does not stop it during a scale-in event. After the task finishes the message, it sets ProtectionEnabled to false so that the task is again eligible for termination. For more information, see Protect your Amazon ECS tasks from being terminated by scale-in events.

Considerations

Consider the following when you configure queue-based scaling for your Amazon ECS service:

  • Because Application Auto Scaling evaluates the metric math expression for you, you do not need to publish a custom metric or run any additional code to compute backlog per task.

  • The RunningTaskCount metric is available only when CloudWatch Container Insights is turned on for your cluster. If it is not, the metric math expression has no data for the divisor and scaling does not occur.

  • Choose a target value based on the latency that your application's service level agreement (SLA) allows. A lower target value keeps queue delay shorter but runs more tasks. A higher target value reduces cost but increases the time that messages wait in the queue.

  • Messages that have not finished processing when a task is stopped return to the Amazon SQS queue. Another running task can process them if you configure the visibility timeout appropriately. To keep a task from being stopped while it is processing a message, use task scale-in protection, as described in Protect tasks that are processing messages.

  • The general considerations for Amazon ECS Service Auto Scaling and target tracking also apply to queue-based scaling.

Configure queue-based scaling

The following steps show how to create a target tracking scaling policy that uses a metric math expression to scale your Amazon ECS service on backlog per task. You use the AWS CLI for these tasks.

  1. Confirm that you have the IAM permissions required to create and update services and to register scalable targets and scaling policies. For more information, see IAM permissions required for Amazon ECS service auto scaling.

  2. Turn on CloudWatch Container Insights for your cluster so that the RunningTaskCount metric is available. For more information, see Monitor Amazon ECS containers using Container Insights with enhanced observability.

  3. Register your Amazon ECS service as a scalable target by using the register-scalable-target command. The scalable dimension is ecs:service:DesiredCount.

    aws application-autoscaling register-scalable-target \
    --service-namespace ecs \
    --scalable-dimension ecs:service:DesiredCount \
    --resource-id service/my-cluster/my-service \
    --min-capacity 1 \
    --max-capacity 10

  4. Store the metric math expression in a customized metric specification in a JSON file named config.json. Replace the queue name, cluster name, and service name with your own values, and set TargetValue to the acceptable backlog per task that you calculated.

    {
    "CustomizedMetricSpecification": {
        "Metrics": [
            {
                "Label": "Get the queue depth",
                "Id": "m1",
                "MetricStat": {
                    "Metric": {
                        "MetricName": "ApproximateNumberOfMessagesVisible",
                        "Namespace": "AWS/SQS",
                        "Dimensions": [
                            { "Name": "QueueName", "Value": "my-queue" }
                        ]
                    },
                    "Stat": "Sum"
                },
                "ReturnData": false
            },
            {
                "Label": "Get the running task count",
                "Id": "m2",
                "MetricStat": {
                    "Metric": {
                        "MetricName": "RunningTaskCount",
                        "Namespace": "ECS/ContainerInsights",
                        "Dimensions": [
                            { "Name": "ClusterName", "Value": "my-cluster" },
                            { "Name": "ServiceName", "Value": "my-service" }
                        ]
                    },
                    "Stat": "Average"
                },
                "ReturnData": false
            },
            {
                "Label": "Calculate the backlog per task",
                "Id": "e1",
                "Expression": "m1 / m2",
                "ReturnData": true
            }
        ]
    },
    "TargetValue": 100
}

  5. Create the target tracking scaling policy by using the put-scaling-policy command with the JSON file as input. Application Auto Scaling creates and manages the two CloudWatch alarms (one for scale-out and one for scale-in) that invoke the policy.

    aws application-autoscaling put-scaling-policy \
    --policy-name sqs-backlog-target-tracking-policy \
    --service-namespace ecs \
    --resource-id service/my-cluster/my-service \
    --scalable-dimension ecs:service:DesiredCount \
    --policy-type TargetTrackingScaling \
    --target-tracking-scaling-policy-configuration file://config.json

For more information about target tracking scaling policies, see Target tracking scaling policies in the Application Auto Scaling User Guide.