How to Choose a CI/CD Tool: Cost Scaling, Languages, and Platforms, and More

How to Choose a CI/CD Tool: Cost Scaling, Languages, and Platforms, and More

How should CI/CD tool cost scaling, language support, and platform support affect your implementation decisions? In a previous post, we looked at the factors you should consider when choosing between a SaaS CI/CD tool vs. a self-hosted CI/CD solution. In this post, we will take a look at a number of other factors that should be considered when evaluating a SaaS CI/CD tool to determine if it’s the right fit for your organization, including cost scalability and language/platform support.

CI/CD Tool Cost Scaling

One thing that is important to keep in mind when deciding to use a paid subscription-based service is how the cost scales with your usage. There are a number of factors that can affect cost. Particularly, some CI/CD SaaS services limit the number of build processes that can be run concurrently. For example, Codeship’s free plan allows only one concurrent build at a time. Travis CI’s travis-ci.org product offers up to 5 concurrent builds for open source projects, but (interestingly) their $69 USD/mo plan on travis-ci.com only offers 1 concurrent build. All of this means that increased throughput will likely result in increased cost. If you expect to maintain a steady level of throughput (that is, you don’t expect to add significantly more developers, which would require additional CI/CD throughput) then perhaps limits on the number of concurrent build processes is not a concern for you. However, if you’re planning on adding more developers to your team, you’ll likely end up having more build/test jobs that need to be executed. Limits may hamper your team’s productivity.

Another restriction you may run across is a limit on the total number of “build minutes” for a given subscription. In other words, the cumulative number of minutes that all build/test processes can run during a given subscription billing cycle (typically a month) is capped at a certain amount. For example, CircleCI’s free plan is limited to 1,500 build minutes per month, while their paid plans offer unlimited build minutes. Adding more developers to your team will likely result in additional build jobs, which will increase the required amount of build minutes per month, which may affect your cost. Additionally, increasing the complexity of your build/test process may result in longer build/test times, which will further increase the number of build minutes you’ll need during each billing cycle. The takeaway here is that if you have a solid understanding of how your team and your build processes are likely to scale in the future, then you should be well equipped to make a decision on whether the cost of a build minute-limited plan will scale adequately to meet your organization’s needs.

Though not directly related to cost scaling, it’s important to note that some CI/CD SaaS providers place a limit on the length of time allowed for any single build/test job, independent of any cumulative per-billing-cycle limitations. For example, Travis CI’s travis-ci.org product limits build jobs to 50 minutes, while jobs on their travis-ci.com product are limited to 120 minutes per build. Similarly, Atlassian’s Bitbucket Pipelines limits builds to 2 hours per job. These limits are probably more than sufficient for most teams, but if you have any long-running build/test processes, you should make sure that your jobs will fit within the time constraints set by your CI/CD provider.

CI/CD Language and Platform Support

Not all languages and platforms are supported by all SaaS CI/CD providers. Support for programming languages, operating systems, containers, and third-party software installation are just a few of the factors that need to be considered when evaluating a SaaS CI/CD tool. If your team requires Microsoft Windows build servers, you are immediately limited to a very small set of options, of which AppVeyor is arguably the most popular. If you need to build and test iOS or Android apps, you have a few more options, such as Travis CI, fastlane, and Bitrise, among others.

Programming languages are another area of consideration. Most providers support the most popular languages, but if you’re using a less popular language, you’ll need to choose carefully. For instance, Travis CI supports a huge list of programming languages, but most other SaaS CI/CD providers support only a handful by comparison. If your project is written in D, Erlang, Rust, or some other less mainstream language, many SaaS CI/CD providers may be a no-go right from the start.

Further consideration is required when dealing with Docker containers. Some SaaS CI/CD providers offer first-class support for Docker containers, while other providers do not support them at all. If Docker is an integral part of your development and build process, some providers may be immediately disqualified from consideration due to this point alone.

Final Thoughts

As you can see, when it comes to determining the CI/CD tool that’s right for your team, there are numerous factors that should be considered, especially with regard to CI/CD tool cost. Fortunately, many SaaS CI/CD providers offer a free version of their service, which gives you the opportunity to test drive the service to ensure that it supports the languages, platforms, and services that your team uses. Just remember to keep cost scaling in mind before making your decision, as the cost of “changing horses” can be expensive should you find that your CI/CD tool cost scales disproportionately with the rest of your business.

In a future post, we will explore third-party integrations with CI/CD tools, with a focus on continuous delivery.

How to Decide Between a CI/CD SaaS Tool vs. Self Hosted

How to Decide Between a CI/CD SaaS Tool vs. Self Hosted

You may find yourself deciding whether to choose a CI/CD SaaS tool, or a self-hosted option. The continuous integration/continuous delivery platform market has grown over the last several years as DevOps becomes more mainstream, and now encompasses a huge variety of tools, each with their own flavor. While it’s great to have choices, it means that choosing the right tool can be a difficult decision. There are several factors to consider when choosing the right fit, including hosting, cost, scalability, and integration support. In this post, we will look at one of the biggest points of consideration: whether to choose a SaaS CI/CD service or a self-hosted system. This will be the first entry in a series of posts about how to choose the CI/CD system that is right for your team. Like everything, there are pros and cons to all solutions, and with the vast amount of CI/CD options available today, there’s no such thing as “one size fits all”.

Considerations for Choosing a CI/CD SaaS Tool

First, let’s take a look at the up-side to choosing a CI/CD SaaS tool. Like most SaaS products, one of the biggest benefits is that there is no hardware or software infrastructure to maintain. There’s no need to worry about server maintenance or applying software updates/patches: that’s all handled for you. In addition to the reduced ongoing maintenance burden, most SaaS CI/CD systems tend to be easy to get set up, especially if you’re using a SaaS VCS (Version Control System) like GitHub or Bitbucket. 

These are great points, but there are potential down-sides that must be considered. The cost of usage for a SaaS CI/CD solution may not scale nicely with your business. For example, the price of a SaaS CI/CD service may go up as your team gets larger. If you plan on scaling your team significantly, the cost of your CI/CD system could inflate dramatically. Furthermore, not all services support all platforms, tools, and environments. If you plan on introducing any new development technologies, you should make sure that they are supported by the CI/CD provider you choose.

Considerations for a Self-Hosted CI/CD Tool

While there are many attractive points in favor of a SaaS CI/CD service, a self-hosted solution is not without its merits. One potential benefit of a self-hosted solution is extensibility. Some self-hosted services can be customized with plugins/extensions to enable functionality that is not included “out of the box”. Jenkins is a prime example of this, with over 1,000 plugins available. Even without plugins, self-hosted CI/CD tools often have more support for development platforms, languages, and testing frameworks than many SaaS solutions. If there’s not first-class support (or a plugin/extension) for a technology that you use, you can usually make things work with some shell scripts and a little bit of creativity. In addition to extensibility, self-hosted solutions typically have fewer limitations on things like build configurations and concurrent build jobs. This isn’t always the case, however. The free version of TeamCity, a CI/CD tool from JetBrains, is limited to 100 build configurations and 3 build agents. Licenses for additional configurations and build agents are available for purchase, though.

Conversely, there are some potential down-sides to a self-hosted system. Perhaps the biggest of these is that you are required to manage your own infrastructure. This includes applying software updates/patches, and may include management of hardware if you’re not hosting the service on an IaaS platform like AWS, GCP, or Azure. In contrast to a SaaS solution, self-hosted systems may require a time-intensive process to get set up. Between getting the system linked up to your VCS (Version Control System), issue/ticket tracking software, and notification system(s), there can be a steep entrypoint in getting your CI/CD system initialized. In addition to the first-time setup, you may be required to manage authentication and authorization for users in your organization if the system you choose doesn’t support your organization’s user management system (LDAP, Google GSuite, etc.).

Final Thoughts

It is worth noting that some CI/CD SaaS tool providers offer self-hosted variants of their services. For instance, CircleCI offers an enterprise solution that can be self-hosted on your own networks, and Travis CI offers Travis CI Enterprise, which is optimized for deployment on Amazon EC2 instances. These offerings throw even more into the mix, and should be part of your consideration when determining which tool has the best fit.

As you can see, there are several factors that must be considered when choosing the CI/CD tool that is right for you. In this post, we discussed some of the trade-offs between SaaS and self-hosted systems. In future posts, we will look at other factors such as scalability, cost, and restrictions/limitations.