Run a Task on Each ECS Cluster Node

If you’re using Amazon ECS with tools such as Datadog or local log shippers, you’ve likely encountered the problem of needing to run an instance of a given container on each ECS cluster machine.

Personally, I’ve needed to run a Datadog (DD-Agent) ECS task on each of my container instances to ensure that Docker-level and APM metrics get sent into Datadog. One might also run a log shipping container on each container instance that ships your application logs to another destination. Prior to a week ago, I was using the spread(instanceId) task placement strategy in ECS to accomplish this, but Amazon recently announced a new feature to ECS which handles this for us in a more effective manner.

For the official Datadog Agent Docker image, you can check out their Docker repository here:

https://hub.docker.com/r/datadog/docker-dd-agent/

Limitations of ECS Daemon Scheduling

It is worth noting that this functionality is singularly available for the EC2 launch type at the time of writing this article.

ECS Daemon Scheduling

In June of 2018, AWS officially released a new way to handle running an instance of an ECS task on each cluster machine; this feature is called “Daemon Scheduling.”

Why Use Daemon Scheduling?

I’m really excited about Daemon Scheduling, as it simplifies things a bit for our DevOps team. Instead of needing to be intimately familiar with how the task placement strategy controls the behavior of the service, one simply needs to note the service type.

How to Leverage Daemon Scheduling

As the underlying autoscaling group that underpins your ECS cluster scales out, the new cluster machines join the cluster. When this happens, AWS will automatically handle running any daemon scheduled tasks on your cluster.

In order to set up this functionality, we simply need to create an ECS service that is configured as a “Daemon” service type. You’ll notice that preexisting services are referred to as “replicas.”

If we were configuring the Datadog agent Docker image, we might name our service dd-agent and set the service type to “daemon”, as seen below:

As you can see, we’re not asked how many instances of our service’s containers we want to run on each cluster node. It’s implied that ECS should only run one of each container in the cluster. Depending on the demands of the service you’re trying to run, one might need to beef up the resources defined in the task definition to ensure the service keeps up with the demands of the cluster.

If you’re not already using Amazon ECS to host some of your applications, I highly recommend it. The benefits of autoscaling in ECS are incredible and have profoundly changed the way I approach architecting systems and applications.

If you’d like quick access to your resources in ECS, check out our Clouductivity Navigator Chrome Extension. We eliminate the clicking and waiting for multiple pages to load, and instead allow you to search from a single pane of glass to get nearly anywhere in the AWS Console. Try it for free today!

--

About the Author

Marcus Bastian is a Senior Site Reliability Engineer that works with AWS and Google Cloud customers to implement, automate, and improve the performance of their applications.

Find his LinkedIn here.

---