**Event-Driven Autoscaling on EKS with KEDA and Amazon SQS**
In event-driven Kubernetes architectures, CPU and memory utilization often fail to reflect real system pressure. A worker pod may sit idle from a CPU perspective while thousands of messages pile up in an Amazon SQS queue. In other cases, pods may continue running long after a traffic spike has passed. For asynchronous, queue-based workloads, backlog is the true scaling signal—not infrastructure utilization.
This article explains how to use Amazon SQS queue depth as an autoscaling metric for event-driven workers, how to deploy KEDA on Amazon EKS with AWS pod identity, and how to tune scaling behavior for reliable, cost-efficient operations.
—
### Why This Matters
In queue-driven systems, delayed message processing directly impacts users and downstream systems. Scaling based on SQS depth aligns autoscaling with actual demand, enabling faster burst handling and lower idle costs when queues are empty.
—
### 1. Prerequisites
Before implementing KEDA-based autoscaling with Amazon SQS, ensure you have:
– An Amazon EKS cluster running a supported Kubernetes version
– KEDA installed in the cluster (for example, via the KEDA Helm chart)
– An AWS authentication method selected (IRSA or EKS Pod Identity)
– An existing Amazon SQS queue
– A worker deployment designed to consume messages from the queue
—
### 2. Architecture and Request Flow
This architecture relies on KEDA observing Amazon SQS queue metrics and managing a Kubernetes Horizontal Pod Autoscaler (HPA) based on backlog.
**Request flow:**
1. Producers send messages to the Amazon SQS queue
2. KEDA polls queue attributes to determine backlog
3. KEDA updates the target metrics in the HPA
4. The HPA scales the worker Deployment
5. Kubernetes schedules additional Pods to process messages
6. As the queue drains, replicas scale back down
This model keeps autoscaling decisions tied directly to outstanding work.
—
### 3. Implementation Steps
#### Install KEDA via Helm
The recommended way to install KEDA is via Helm:
“`bash
# Add the KEDA Helm repository
helm repo add kedacore
helm repo update
# Install KEDA into a dedicated namespace
helm install keda kedacore/keda
–namespace keda
–create-namespace
“`
*What this does:* Deploys the KEDA operator and metrics API server, along with CRDs such as ScaledObject and TriggerAuthentication.
#### Deploy the Worker with AWS Identity Attached
An SQS consumer typically requires permissions such as:
– GetQueueAttributes
– GetQueueUrl
– ReceiveMessage
– DeleteMessage
– ChangeMessageVisibility
**Example IAM policy:**
“`json
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Sid”: “ConsumeFromQueue”,
“Effect”: “Allow”,
“Action”: [
“sqs:GetQueueAttributes”,
“sqs:GetQueueUrl”,
“sqs:ReceiveMessage”,
“sqs:DeleteMessage”,
“sqs:ChangeMessageVisibility”
],
“Resource”: “arn:aws:sqs:us-east-1:123456789012:orders-queue”
}
]
}
“`
*What this does:* Enforces least-privilege access by granting permissions only for the required SQS queue.
#### Configure KEDA Authentication and ScaledObject
KEDA connects the deployment to the SQS queue using TriggerAuthentication and ScaledObject. Using `queueURLFromEnv` avoids hardcoding the queue URL.
“`yaml
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: sqs-processor-auth
namespace: workers
spec:
podIdentity:
provider: aws
—
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: sqs-processor
namespace: workers
spec:
scaleTargetRef:
name: sqs-processor
pollingInterval: 10
cooldownPeriod: 120
minReplicaCount: 0
maxReplicaCount: 30
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 60
triggers:
– type: aws-sqs-queue
authenticationRef:
name: sqs-processor-auth
metadata:
queueURLFromEnv: QUEUE_URL
awsRegion: us-east-1
queueLength: “10”
activationQueueLength: “1”
“`
*What this does:* Defines how KEDA authenticates with AWS and how replicas are calculated. Each pod targets 10 messages, and scale-to-zero is enabled when the queue is empty.
—
### 4. Replica Calculation Logic
KEDA calculates outstanding work as:
`ApproximateNumberOfMessages + ApproximateNumberOfMessagesNotVisible`
(+ delayed messages, if enabled)
**Replica calculation:**
`desired replicas = ceil(outstanding messages / queueLength)`
**Example (queueLength: “10”):**
– 0 messages → 0 pods
– 8 messages → 1 pod
– 25 messages → 3 pods
– 95 messages → 10 pods
Final replica counts are bounded by `minReplicaCount` and `maxReplicaCount`.
—
### 5. Verification
After applying the manifests:
1. Send 50 messages to the SQS queue
2. Inspect the ScaledObject: `kubectl get scaledobject sqs-processor -n workers`
3. Watch pod scaling: `kubectl get pods -n workers -w`
4. Confirm pods scale back to zero after processing completes
—
### 6. Troubleshooting Common Issues
– **No scale-out**
– Cause: Incorrect queue URL or IAM permissions
– Check: `kubectl describe scaledobject`, KEDA operator logs
– **Too many pods**
– Cause: In-flight messages counted
– Check: `scaleOnInFlight`, SQS visibility timeout
– **Never scales to zero**
– Cause: Delayed or unacknowledged messages
– Check: Queue attributes and application behavior
– **HPA exists but no scaling**
– Cause: Authentication or metric fetch errors
– Check: KEDA logs and fallback status
—
### 7. Production Tuning Tips
– Choose `queueLength` based on throughput, not guesswork
– Tune cooldown and HPA behavior separately—they control different scale-down paths
– Enable fallback replicas for resilience if metrics become unavailable
– Decide whether in-flight messages should count based on visibility timeout behavior
—
### Conclusion
Queue-based workloads benefit most when autoscaling is driven by the amount of work waiting to be processed rather than traditional infrastructure metrics. Backlog-aware scaling enables applications to react more quickly to changing demand while reducing unnecessary resource consumption during quieter periods.
Although this article demonstrated the approach using Amazon SQS, Amazon EKS, and KEDA, the same design pattern applies across many event-driven architectures and messaging platforms. The key is selecting a scaling signal that accurately represents workload demand and tuning the autoscaling behaviour to match the characteristics of the application.
As organisations increasingly adopt asynchronous, event-driven systems, workload-aware autoscaling becomes an important part of building resilient, efficient, and cost-effective Kubernetes deployments.
—
## FAQ
**What is backlog-based autoscaling?**
Backlog-based autoscaling uses the number of pending messages in a queue (backlog) as the primary metric for scaling workers, rather than CPU or memory usage. This ensures that scaling reflects actual work waiting to be processed.
**Which AWS services are involved?**
The main AWS services are Amazon SQS (queue) and Amazon EKS (Kubernetes cluster). IAM handles authentication and permissions.
**What is KEDA?**
KEDA (Kubernetes Event-driven Autoscaling) is an open-source Kubernetes operator that enables event-driven scaling. It connects to external triggers like SQS and adjusts Kubernetes replica counts accordingly.
**How is queue depth calculated in KEDA?**
KEDA sums `ApproximateNumberOfMessages` and `ApproximateNumberOfMessagesNotVisible` from SQS queue attributes, then divides by `queueLength` and rounds up to determine desired replicas.
**What are activationQueueLength and queueLength?**
– `queueLength`: Target number of messages per pod. When backlog exceeds `queueLength * replicas`, scaling occurs.
– `activationQueueLength`: Minimum backlog required to trigger scaling. Until this threshold is met, no replicas are created (when `minReplicaCount` is 0).
**What are cooldown periods?**
Cooldowns control how long KEDA waits before scaling down after the queue drains (`cooldownPeriod`) and how often scaling checks occur (`pollingInterval`).
**How does HPA behavior affect scaling?**
HPA behavior settings (`scaleDown.stabilizationWindowSeconds`) prevent rapid scale-up and scale-down oscillations by applying time-based windows around scaling decisions.
**Can I use this pattern with other message brokers?**
Yes. KEDA supports many triggers (RabbitMQ, Kafka, Azure Queue Storage, etc.). The same principles apply: use queue depth/backlog as the scaling metric.
**Do I need pod identity for AWS authentication?**
Yes. KEDA requires either IRSA (IAM Roles for Service Accounts) or EKS Pod Identity to securely access AWS resources without embedding credentials.
**How do I verify scaling works?**
Send a known number of messages to the queue, monitor pod count with `kubectl get pods`, and check ScaledObject metrics with `kubectl get scaledobject`.
—
