AWS

🔧 Fixing AWS Beanstalk → CloudWatch Log Streaming for Sidekiq Logs

Dai Dinh

Dai Dinh

— 3 min read
🔧 Fixing AWS Beanstalk → CloudWatch Log Streaming for Sidekiq Logs
Photo by Jordan Harrison / Unsplash

Streaming logs from your AWS Elastic Beanstalk app to CloudWatch should be easy. But as I recently discovered while working on a test deployment, getting Sidekiq logs to show up can be trickier than expected—especially when permissions and config don’t line up.

Here’s what I learned, the issue I ran into, and how I fixed it.

🧠 The Problem

I had a Rails app using Sidekiq for background jobs, deployed on:

  • AWS Elastic Beanstalk (Amazon Linux 2023)
  • Sidekiq logs written to: /var/app/current/log/sidekiq.log
  • CloudWatch logs enabled from the Beanstalk console

I could already see logs like:

/aws/elasticbeanstalk/<env-name>/var/log/web.stdout.log
/aws/elasticbeanstalk/<env-name>/var/log/puma/puma.log

But sidekiq.log ** never showed up**.

❌ First Mistake: Malformed Config

I initially created a .ebextensions config to push logs via the CloudWatch Agent, but deployment failed with:

****** processing amazon-cloudwatch-agent ******
E! downloadLocation json is malformated.

Cause:

👉 CloudWatch Agent config must be strict JSON — YAML or comments won’t work.

🔐 Hidden Problem: IAM Permissions

Even after fixing the JSON, I still couldn’t see the logs. The real issue?

Elastic Beanstalk's EC2 instance profile didn’t have permission to write logs to CloudWatch.

✅ Solution: Update IAM Policy

Here’s the IAM policy I attached to the Beanstalk EC2 role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": "*" # configuration your resource
    }
  ]
}

Without this, the CloudWatch Agent has no right to create or send logs—even if your config is perfect.

✅ Final Working Config

Here’s the .ebextensions config that finally worked:

.ebextensions/01-cloudwatch-logs.config

files:
  "/tmp/custom-cloudwatch-config.json":
    mode: "000600"
    owner: root
    group: root
    content:
      Fn::Sub: |
        {
          "logs": {
            "logs_collected": {
              "files": {
                "collect_list": [
                  {
                    "file_path": "/var/app/current/log/sidekiq.log",
                    "log_group_name": "/aws/elasticbeanstalk/${AWSEBEnvironmentName}/sidekiq.log",
                    "log_stream_name": "{instance_id}"
                  }
                ]
              }
            }
          }
        }

container_commands:
  01_append_sidekiq_logs_config:
    command: |
      /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
        -a append-config \
        -m ec2 \
        -c file:/tmp/custom-cloudwatch-config.json \
        -s

Why it works:

  • Fn::Sub injects the environment name dynamically
  • append-config keeps Beanstalk’s default log settings intact
  • Secure permissions (0600) prevent exposure

✅ Debugging Checklist

To troubleshoot why your Sidekiq log is not appearing in CloudWatch, follow these steps based on AWS documentation:

Here’s a step-by-step checklist to debug CloudWatch logs:

  1. ✅ Check CloudWatch Agent Logs
    SSH into your EC2 instance and inspect the CloudWatch Agent logs:
sudo cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Look for errors related to config loading or file permissions.

2. ✅ Verify File Existence and Permissions
Ensure the Sidekiq log file exists and is readable:

ls -l /var/app/current/log/sidekiq.log
sudo cat /var/app/current/log/sidekiq.log
  1. ✅ Validate Config File
    Ensure /tmp/custom-cloudwatch-config.json is created and contains the expected JSON structure.
  2. ✅ Check CloudWatch Log Group/Stream
    In the AWS Console, navigate to CloudWatch > Log groups and confirm that /aws/elasticbeanstalk/<your-env-name>/sidekiq.log exists and has recent log streams.
  3. ✅ Review Documentation
    Reference the AWS Docs on customizing CloudWatch Logs for deeper troubleshooting.
  4. ✅ Restart the CloudWatch Agent
    Sometimes a manual restart helps:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start

🎉 Result

After making both fixes (valid config + proper IAM permissions), Sidekiq logs started streaming to:

/aws/elasticbeanstalk/<env-name>/sidekiq.log

Now I can tail Sidekiq jobs from CloudWatch instead of SSH'ing into EC2. Much better.

💡 Final Thoughts

This was a small but valuable lesson:

Even if your configuration is perfect, permissions can silently block everything.

If you're deploying a background job system on Beanstalk, make sure logging is set up properly—especially if you're debugging production behavior.

🚀 Want More Real-World DevOps & Indie Dev Tips?

I share lessons learned while building and scaling apps as a solo dev, from AWS to Rails to side projects.

👉 Visit my blog at blog.daidtech.com for more posts like this.
👉 See my full portfolio at daidtech.com

Let’s build smarter — together.

Read more