If you use AWS, you may need an AWS job scheduler to automate planned and repeating tasks. For example, system backups, monitoring system status, sending notifications to end users or other systems, data analytics reports, deleting unnecessary resources, etc.
Whether you’re using cron jobs or another system, let’s take a look at a few options to create cron-job-like scheduled tasks, and a few examples of repeating tasks that can help reduce costs.
Options to Create AWS Scheduled Tasks
Given the simplicity of Lambda, many AWS users prefer it to schedule jobs. You’ll need to create rules to trigger your Lambda function, for example, a CloudWatch rule such as “run at 10:00 a.m. every day” or “run at 8:00 a.m. every 1st day of the month.” This option is best if the job will complete within 15 minutes and can be created as a Lambda.
Another method for longer or slightly more complex jobs is the same, but using the CloudWatch rule to start a Lambda that then launches an EC2 instance to run a script using the AWS “user data” option. If you use this method, make sure the last thing the script does is to terminate the instance. Otherwise, you’ll keep racking up costs on an EC2 instance you’re not using, resulting in wasted spend.
You can also use ECS scheduled tasks if your code is packaged as a Docker container. This is a good option if your job needs longer than the 15-minute limitation of Lambda. You can schedule it with a CloudWatch rule as above, or via the “scheduled tasks” option in the default cluster in AWS. You will need a Docker image in Amazon ECR or Docker Hub, then use a task definition in Amazon ECS to define a launch type of Fargate, as well as the image, environment, command, etc.
You can also use AWS batch jobs for more complex jobs, which offers the ability to run multiple jobs in parallel, re-run failed jobs, etc.
How to Use AWS Scheduled Events to Reduce Resource Costs
How does this help reduce costs? Well, AWS CTO Werner Vogels has said that “a core principle from Lean is to eliminate waste. Waste is anything that doesn’t benefit your customers.”
Vogels has a recommendation to eliminate costs -“One way to save really significant dollars in dev and test is to switch your resources off when you go home. Typically you can save up to 75% on their dev and test costs just by switching resources off when you go home.”
We have calculated that approximately $11 billion will be wasted on idle non-production resources left running when not being used, in 2020 alone.
You can use your choice of scheduled events to turn non-production resources off at the end of the workday. For example, all EC2 instances tagged “dev” could be turned off from 7 PM to 7 AM and on weekends, resulting in a 65% cost reduction.
Things Can Get Messy
However, once you start creating rules to turn these resources off, you’ve essentially created an app that now has overhead to maintain. How will you add it to new resources? What if users need their resources outside of normal hours? Who is in charge of ensuring that all resources are in the desired state? Only accessible by the desired people? Are resources across regions and resource types being scheduled?
And once you start looking for ways to reduce wasted spend, they often find that scheduling isn’t enough. They also want ways to rightsize resources to match the necessary capacity. So, scheduled jobs soon aren’t enough.
The ParkMyCloud platform was created to automate resource on/off scheduling, as an alternative to labor-intensive methods like event scheduling. It can assign schedules automatically, provides user governance, has a simple schedule override method, provides an easy-to-use UI, covers multiple accounts and cloud providers, offers robust reporting – and importantly, goes beyond simply scheduling. With your utilization data as a basis, ParkMyCloud discovers resources that can be rightsized, as well as unnecessary volumes/snapshots, and more. See a full comparison of ParkMyCloud vs. in-house developed solutions.