Among the various ways to lose money on idle or orphaned resources in the cloud, here’s another for AWS users to add to the list: unused reserved Instances. At first, the idea of wasting money on AWS Reserved Instances seems counterintuitive. After all, aren’t RIs meant to save money? The short answer is yes – but only if you use them efficiently.
How Unused Reserved Instances Occur
To understand how unused Reserved Instances contribute to cloud waste, consider how they work. With AWS Reserved Instances, you’re making a commitment of usage by renting instances for a fixed amount of time in exchange for a lower rate (per-hour or per-second) than on-demand. You’re still free to use all the same families, OS types, and instance sizes with either one, except with RIs your ability to use certain instance types is limited to the purchasing plan you choose.
The only real difference between an AWS On-Demand instance and an AWS Reserved Instance is how you get billed for them on the backend – and this is where it gets tricky. You don’t know if your Reserved Instances have been used until you get the bill. Instead, you run your instances as you always would, with no insight into what will get billed as reserved instances. It’s only when your bill is created the following month that AWS reviews your reservations alongside your usage to apply the Reserved Instances that match up with your workload. This leaves you with little visibility into what your costs will be, forcing you to track usage on your own, and running the risk of unused reservations that result in, you guessed it – wasted money.
Ways to Avoid Losing Money Unused Reserved Instances
Reserved Instances require commitment of usage, ongoing awareness and insight into your future costs, and the possibility of going unused if AWS can’t apply them sufficiently. But that doesn’t mean you should shy away from using them. Reserved Instances can be cost-effective if used with a few things in mind:
Pick the RI type that suits your usage and workload. The best plan is one of prevention. Before you get started with purchasing reservations, get a detailed look at your usage and the most optimal instance types for your workload (something you should already be doing as part of your cost control measures). By design, Reserved Instances work best with steady state workloads and consistent usage. Once you confirm that your usage makes you a good candidate, you’ll want to choose the RI instance type that will benefit your needs most:
- Standard RIs – Recommended for steady-state usage, and provide the most savings.
- Convertible RIs – a smaller discount from On-Demand instances, but in return provide flexibility to change families, OS types, and tenancies.
- Scheduled RIs – similar to Standard RIs, but only apply to instances launched within the time windows you select, which can recur on a daily, weekly, or monthly schedule.
Sell unused reserved instances on the Reserved Instance Marketplace. Using the marketplace allows you to list your reservation for purchase by other users. The cheapest reservations are sold first, and once someone purchases yours, you’ll be the charged the on-demand rate whenever you use that instance type moving forward.
Purchase convertible reservations. With convertible reservations, you have the option to convert your reserved instances to other types, so long as the new type is more expensive. You won’t get as much of a discount, but flexibility and more options for use make up for the smaller savings.
The Lesson to be Learned
Just like any other idle or unused cloud resource, unused reserved instances can only do one thing – waste your money. Cloud services were meant to help you keep infrastructure costs in check, but only if you use them smartly. Does all this sound complex to manage? That’s because it is. Take the DIY route by keeping an eye on your infrastructure, or put our automated solution to work and check out our Reserved Instance manager.
In our ongoing discussion on cloud waste, we recently talked about orphaned resources eating away at your cloud budget, but there’s another type of resource that’s costing you money needlessly and this one is hidden in plain sight – overprovisioned resources. When you looked at your initial budget and made your selection of cloud services, you probably had some idea of what resources you needed and in what sizes. Now that you’re well into your usage, have you taken the time to look at those metrics and analyze whether or not you’ve overprovisioned?
One of the easiest ways to waste money is by paying for more than you need and not realizing it. Here are 6 types of overprovisioned resources that contribute to cloud waste.
As a rule of thumb, it’s a good idea to delete volumes that are not attached to instances or VMs. Take the example of AWS EBS volumes unattached to EC2 instances – if you’re not using them, then all they’re doing is needlessly accruing charges on your monthly bill. And even if your volume is attached to an instance, it’s billed separately, so you should also make a practice of deleting volumes you no longer need (after you backup the data, of course).
Underutilized database warehouses
Data warehouses like Amazon Redshift, Google Cloud Datastore, and Microsoft Azure SQL Data Warehouse were designed as a simple and cost-effective way to analyze data using standard SQL and your existing Business Intelligence (BI) tools. But to get the most cost savings benefits, you’ll want to identify any clusters that appear to be underutilized and rightsize them to lower costs on your monthly bill.
Underutilized relational databases
Relational databases such as Amazon RDS, Azure SQL, and Google Cloud SQL offer the ability to directly run and manage a relational database without managing the infrastructure that the database is running on or having to worry about patching of the database software itself.
As a best practice, Amazon recommends that you check the configuration of your RDS for any idle DB instances. You should consider a DB instance idle if it has not had a connection for a prolonged period of time, and proceed by deleting the instance to avoid unnecessary charges. If you need to keep storage for data on the instance, there are other cost-effective alternatives to deleting altogether, like taking snapshots. But remember – manual snapshots are retained, taking up storage and costing you money until you delete them.
We often preach about idle instances and how they waste money, but sizing your instances incorrectly is just as detrimental to your monthly bill. It’s easy to overspend on large instances or VMs that are you don’t need. With any cloud service, whether it’s AWS, Azure, or GCP, you should always “rightsize” your instances and VMs by picking the instance size that is optimized for the size of your workload – be it compute optimized, memory optimized, GPU optimized, or storage optimized.
Once your instance has been running for some time, you’ll have a better idea of whether not the chosen size is optimal. Review your usage and make cost estimates with AWS Management Console, Amazon CloudWatch, and AWS Trusted Advisor if you’re using AWS. Azure users can review their metrics from Azure Monitor data, and Google users can import GCP metrics data for GCP virtual machines. Use this information to find under-utilized resources that can be resized to better optimize costs
Application containerization allows multiple applications to be distributed across a single host operating system without requiring their own VM, which can lead to significant cost savings. It’s possible that developers will launch multiple containers and fail to terminate them when they are no longer required, wasting money. Due to the number of containers being launched compared to VMs, it will not take long for container-related cloud waste to match that of VM-related cloud waste.
The problem with controlling cloud spend using cloud management software is that many solutions fail to identify unused containers because the solutions are host-centric rather than role-centric.
Idle hosted caching tools (Redis)
Hosted caching tools like Amazon ElastiCache offer high performance, scalable, and cost-effective caching. ElastiCache also supports Redis, an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. While caching tools are highly useful and can save money, it’s important to identify idle cluster nodes and delete them from your account to avoid accruing charges on your monthly bill. Be cognizant of average CPU utilization and get into the practice of deleting the node if your average utilization is under designated minimum criteria that you set.
How to Combat Overprovisioned Resources (and lower your cloud costs)
Now that you have a good idea of ways you could be overprovisioning your cloud resources and needlessly running up your cloud bill – what can you do about it? The end-all-be-all answer is “be vigilant.” The only way to be sure that your resources are cost-optimal is with constant monitoring of your resources and usage metrics. Luckily, optimization tools can help you identify and automate some of these best practices and do a lot of the work for you, saving time and money.
This AWS Reserved Instances FAQ is a culmination of questions we are often asked about using Reserved Instances to save on your Amazon Web Services (AWS) costs.
First, let’s make sure you’re all on the same page. AWS Reserved Instances are a way to save money on your cloud bill by reserving capacity in advance, which you can choose to pay for all, some, or no upfront. Due to the different billing model and variety of options, these often raise questions. Let’s explore a few:
Q1: How do I know which instances are reserved?
Because of how AWS billing works, Reserved Instances aren’t applied until you’re actually being charged for the instances. Think of it more like a “voucher” than a specific virtual machine.
This means that you run your instances as you normally would throughout the month, without seeing which will be billed as reserved vs. on-demand. When your bill comes on the 5th of the next month, AWS will look at your reservations and your usage and automatically apply the reservations that match the instances.
This means that you don’t have to decide which instances are reserved, but it also means you don’t have the visibility into what your true costs are going to be. You’ll have to track this on your own, or use some tools to help you manage your Reserved Instances. AWS has included some reports in the Cost Explorer that can help you visualize your month-to-date usage of your reservations.
Q2: What if I don’t need that instance type anymore?
There’s a couple of options for unused reservations, depending on the reason you aren’t using them anymore. The first option is to sell the reservation on the Reserved Instance Marketplace. This marketplace lets you list your reservation and allows others to purchase it from you. The reservations are grouped together, with the cheapest being sold first in that group. Once someone purchases your reservation, you start getting charged at the on-demand rate (if you still use that type).
The other option is to purchase convertible reservations, which allow you to change the attributes of the reservation (as long as it’s to a more expensive instance type). These convertible types give you much more flexibility, especially as your company or application are growing in use, but they do come with a smaller savings rate over standard Reserved Instances.
Q3: How do Reserved Instances work if I have multiple AWS accounts?
Many users of AWS in large organizations have started using a multi-account strategy, either for security reasons or billing reasons. When managing multiple AWS accounts, billing and reservations can get confusing. The good news is that reservations can “float” across multiple linked accounts, even if they weren’t purchased in the master account. Amazon is smart enough to attempt to apply the reservation to the account it was purchased in, but if a suitable instance is not found, it can apply it to an instance in a different account that matches. You can also choose to exclude linked accounts from receiving these floating reservations if you’d like.
Q4: Are AWS Reserved Instances better than On Demand?
Great question. This is actually the subject of the most popular post on the ParkMyCloud blog. In short: usually yes, but only for production environments. Otherwise, On Demand instances with on/off schedules are typically better value.
Reserved Instances can help save a lot of money on your AWS bill, but require a different way of thinking about your instances than the normal pay-as-you-go that you may be used to with cloud computing. These reservations are typically used for steady-state workloads with consistent usage. They can be complex to manage – be sure to check out our Reserved Instances optimizer to get the maximum value out of your reservations.
Hitachi ID Systems recently reached their first ParkMyCloud birthday – to celebrate, we chatted with Patrick McMaster about how they optimize training infrastructure and why he and his team said “We honestly couldn’t be happier with ParkMyCloud.”
Can you start by telling me about Hitachi ID Systems and what you and your team do within the company?
Hitachi ID Systems makes identity and access management (IAM) software. I am the training coordinator, so I handle getting clients and potential partners up to speed with how our software works, how to install it, how to administrate it, etc. For those who are more interested in learning about software, we set them up with a virtual environment and course materials or an instructor to get them up to speed with how the software works.
Can you describe how you’re using public cloud?
We use AWS exclusively. When we advertise that we’re running a course a few months ahead of time, our infrastructure starts seeing the registrations and will start creating VMs, applying patches, getting the latest version of the appropriate software installers on the desktop and getting everything ready for the students, who will be accessing geographically-local AWS infrastructure.
In the past, everything for this online training was very manual. We on the training team would spin up the VMs manually, do the updates manually, and send the information to the potential students., Then when the course was over we would go through and do the reverse – shutting the elements down and turning off the virtual machine on AWS.
What does the supporting training infrastructure in AWS look like?
We have a number of VMs running per student or team that only need to be active during the team’s local business hours, plus some additional supporting infrastructure which is required 24/7. We started to realize as we got more students and began offering self-paced training that our AWS fees were increasing from the 24/7 access we were providing, but also just the management of keeping track of which students are where, when they should be brought into the system, when they should be shut down, etc. We needed to find a solution pretty quickly as we experienced that period of rapid growth.
How did that lead you to finding ParkMyCloud?
We knew we needed to automate the manual processes for this. Of course, lots of organizations tend to come up with solutions internally first. We’re a software company, so we had the talent for that, but we never have enough time. I’ve come to terms more and more every year with the benefits of delegating to the other sources. I realized that we are probably not the first organization to have this problem, so I Googled and found ParkMyCloud.
It became quickly evident that the features that you offered were exactly what we were looking for.
Can you describe your experience as a ParkMyCloud user so far?
Sure! So just before our demo of ParkMyCloud, we were fighting with this issue of trying to figure out how we can manage multiple time zones and multiple geographic locations, and not pay for that time that VMs are just spun up.
Then we went through the ParkMyCloud demo process and started our trial. We connected to our system and looked at ways to set up different schedules and pull information from AWS. There was definitely a moment where everyone in the room looked each other and said, “we must be missing something” – there had to be some additional steps we hadn’t thought out because it seemed too easy to work. But it really was that easy.
It just took a week of monitoring to make sure everything was turning off when it was supposed to – the bulk of our effort was really in that first week, and the time we need to spend in the interface is so small. We can go into ParkMyCloud’s dashboard and make exceptions to the schedule when needed, but the time that we actually spend thinking about these things is now about 2 hours a week, whereas before it was something that members of our staff might struggle with for 1-2 days. It’s been a huge improvement.
We did some calculations just in terms of uptime versus what we were doing before, and having the different schedules at our disposal and being able to spend that one week coming up with every scenario we could come up with was time well-spent. Now there are very few exceptions. I don’t think we’ve had to create a new schedule in a long time. Everything is organized logically, it’s very easy for us to find everything we need.
Who is responsible for tracking your AWS spending in the organization? Have they had any feedback?
Our finance department. Since we started using ParkMyCloud, it’s been very very quiet. No news is good news from finance. We are saving about 40% of our bill.
Do you have any other cloud cost savings measures in place?
Not for this training infrastructure. We have a pretty unique use case here. Our next steps are going to be more towards automatic termination, automatic spinning things up, more time-saving measures rather than cost.
This summer ParkMyCloud is working on instance rightsizing, if that’s something that would be helpful for you.
That’s definitely something that we could use. We are always trying to find better ways of doing things.
OK, great, thanks Patrick! Appreciate your time.
Thank you, have a good one!
We recently discussed how orphaned volumes and snapshots contribute to cloud waste and what you can do about it, but those are just two examples of orphaned cloud resources that result in unnecessary charges. The public cloud is a pay-as-you-go utility, requiring full visibility of specific infrastructure – you don’t want to be charged for resources you aren’t using. Here are other types of orphaned cloud resources that contribute to cloud waste (and cost you money):
Unassociated Elastic IPs
Elastic IPs are reserved public IP addresses designed for dynamic cloud computing in AWS. As a static IPv4 address associated with your AWS account, Elastic IPs can continue running an EC2 instance, even if it is stopped and restarted, by quickly remapping the address to another one of your instances. You can allocate an Elastic IP address to any EC2 instance in a given region, until you decide to release it.
The advantage of having an Elastic IP (EIP) is the ability to mask the failure of an EC2 instance, but if you do not associate the address to your account – you’re still getting charged. To avoid incurring a needless hourly charge from AWS, remember to release any unassociated IPs you no longer need.
Elastic Load Balancers (with no instances)
Cloud load balancing allows users to distribute workloads and traffic with the benefit of the cloud’s scalability. All major cloud providers offer some type of load balancing – AWS users can balance workloads and distribute traffic among EC2 instances with its Elastic Load Balancer, Google Cloud can distribute traffic between VM instances with Google Cloud Load Balancing, and Azure’s Load Balancer distributes traffic across multiple data centers.
An AWS Elastic Load Balancer (ELB) will incur charges to your bill as long as it’s configured with your account. Like with Elastic IPs, whether you’re using it or not – you’re paying. If you have no instances associated with your ELB, delete to avoid paying needless charges on your monthly bill.
Unused Machine Images (AMIs)
A Machine Image provides the information required to launch an instance, which is a virtual server in the cloud. In AWS they’re called AMIs, in Azure they’re Managed Images, and in Google Cloud Platform they’re Custom Images.
As part of your measures to reduce unnecessary costs from orphaned volumes, delete unused machine images when you no longer need them. Unless deleted, the snapshot that was created when the image was first created will continue to incur storage costs.
One of the growing pains that organizations face is the management of isolated pools of data in their cloud environment. Fragmented storage can result from data coming from a number of sources used by applications and business processes. Object Storage was designed to break down silos into scalable, cost-effective storage to store data in its native format. AWS offers object storage solutions like Amazon S3 and Amazon Glacier, Google has Google Cloud Storage, and Azure calls its solution Azure Blob Storage. All options help you manage your storage in one place, keeping your data organized and your business more cost effective.
Although object storage in and of itself is a cost effective solution, there are still ways to optimize and reduce costs within this solution. Delete files you no longer need so that you’re not paying for them. Delete unused files which can be recreated. In S3, use the “lifecycle” feature to delete or overwrite older versions of data. Clean up incomplete uploads that were interrupted, resulting in partial objects that are taking up needless space. And compress your data before storing to give you better performance and reduce your storage requirements.
How to Avoid Wasted Spend on Orphaned Cloud Resources
Don’t let forgotten resources waste money on your cloud bill. Put a stop to cloud waste by eliminating orphaned cloud resources and inactive storage, saving space, time, and money in the process. Remember to:
- Release unassociated IPs you no longer need.
- Remove Elastic Load Balancers with no instances attached.
- Delete unused machine images when you no longer need them.
- Keep object storage minimal – optimize by “cleaning up” regularly, removing files you don’t need.
The premise of the cloud and the resources available to you were meant to be cost effective, but it’s up to you keep costs in check. Optimize your cloud spend with visibility, management, and cost automation tools like ParkMyCloud to get the most out of your cloud environment.
Given that spring is very much in the air – at least it is here in Northern Virginia – our attention has turned to tidying up the yard and getting things in good shape for summer. While things are not so seasonally-focused in the world of cloud, the metaphor of taking time out to clean things up applies to unused cloud resources as well. We have even seen some call this ‘cloud pruning’ (not to be confused with the Japanese gardening method).
Cloud pruning is important for improving both cost and performance of your infrastructure. So what are some of the ways you can go about cleaning up, optimizing, and ensuring that our cloud environments are in great shape?
Delete Old Snapshots
Let’s start with focusing on items that we no longer need. One of the most common types of unused cloud resources is old Snapshots. These are your old EBS volumes on AWS, your storage disks (blobs) on Azure, and persistent disks on GCP. If you have had some form of backup strategy then it’s likely that you will understand the need to manage the number of snapshots you keep for a particular volume, and the need to delete older, unneeded snapshots. Cleaning these up immediately helps save on your storage costs and there are a number of best practices documenting how to streamline this process as well as a number of free and paid-for tools to help support this process.
Delete Old Machine Images
A Machine Image provides the information required to launch an instance, which is a virtual server in the cloud. In AWS these are called AMIs, in Azure they’re called Managed Images, and in GCP Custom Images. When these images are no longer needed, it is possible to deregister them. However, depending on your configuration you are likely to continue to incur costs, as typically the snapshot that was created when the image was first created will continue to incur storage costs. Therefore, if you are finished with an AMI, be sure to ensure that you also delete its accompanying snapshot. Managing your old AMIs does require work, but there are a number of methods to streamline these processes made available both by the cloud providers as well as third-party vendors to manage this type of unused cloud resources.
With the widespread adoption of containers in the last few years and much of the focus on their specific benefits, few have paid attention to ensuring these containers are optimized for performance and cost. One of the most effective ways to maximize the benefits of containers is to host multiple containerized application workloads within a single larger instance (typically large or x-large VM) rather than on a number of smaller, separate VMs. In particular, this is something you could utilize in your dev and test environments rather than in production, where you may just have one machine available to deploy to. As containerization continues to evolve, services such as AWS’s Fargate are enabling much more control of the resources required to run your containers beyond what is available today using traditional VMs. In particular, the ability to specify the exact CPU and memory your code requires (and thus the amount you pay) scales exactly with how many containers you are running.
So alongside pruning your trees or sweeping your deck and taking care of your outside spaces this spring, remember to take a look around your cloud environment and look for opportunities to remove unused cloud resources to optimize not only for cost, but also performance.