As we continue to evaluate ways to automate various aspects of software development, today we’ll take a look at Google Cloud Composer. This is a fully managed workflow orchestration service built on Apache Airflow that makes workflow management and creation simple and consistent.
The evolution of hybrid and multi-cloud environments continue to grow as enterprises want to take advantage of the cloud’s scalability, flexibility, and global reach. Of the three major providers, Google Cloud has been the most open to supporting this multi-cloud reality. For example, earlier this year, Google launched Anthos, a new managed service offering for hybrid and multi-cloud environments to give enterprises operational consistency by running quickly on any existing hardware, leverage open APIs and give developers the freedom to modernize. But, implementing the management of these environments can be either an invaluable proposition for your company or one to completely challenge your infrastructure instead – which brings us to Google’s solution, Cloud Composer.
How does Google Cloud Composer work?
With Cloud Composer, you can monitor, schedule and manage workflows across your hybrid and multi-cloud environment. Here is how:
- As part of Google Cloud Platform (GCP), Cloud Composer integrates with tools like BigQuery, Dataflow, Dataproc, Datastore, Cloud Storage, Pub/Sub and Cloud ML Engine, giving users the ability to orchestrate end-to-end GCP workloads.
- You can code directed acyclic graphs (DAGs) using Python to improve workflow readability and pinpoint areas in need of assistance.
- It has one-click deployment built-in to give you instant and easy access to a range of connectors and graphical representations that show your workflow in action.
- Cloud Composer allows you to pull workflows together from wherever they live, supporting a fully-functioning and connected cloud environment.
- Since Cloud Composer is built on Apache Airflow – an open-source technology – it provides freedom from vendor lock-in as well as integration with a wide variety of platforms.
Simplifying hybrid and multi-cloud environment management
Cloud Composer is ideal for hybrid and multi-cloud management because it’s built on Apache Airflow and operated with the Python programming language. Using open-source technology and the “no lock-in” approach and portability gives users the flexibility to create and deploy workflows seamlessly across clouds for a unified data environment.
Setting up your environment is quick and simple. Pipelines created with Cloud Composer will be configured as DAGs with easy integration for any required Python libraries, giving users of almost any level the ability to create and schedule their own workflows. With the built-in one-click deployment, you get instant and easy access to a range of connectors and graphical representations that show your workflow in action.
However, costs can be a drawback to making the most of your cloud environment when using Cloud Composer. Landing on specific costs for Cloud Composer can be hard to calculate, as Google measures the resources your deployments use and add the total cost of your Apache Airflow deployments onto your wider GCP bill.
Cloud Composer Pricing
Pricing for Cloud Composer is based on the size of a Cloud Composer environment and the duration the environment runs, so you pay for what you use, as measured by vCPU/hour, GB/month, and GB transferred/month. Google offers multiple pricing units for Cloud Composer because it uses several GCP products as building blocks. You can also use the Google Cloud Platform pricing calculator to estimate the cost of using Cloud Composer.
So, should you use Google Cloud Composer? Cloud Composer environments are meant to be long-running compute resources that are always online so that you can schedule repeating workflows whenever necessary. Unfortunately, since you can’t turn on and off a Cloud Composer environment; you can only create or destroy, it may not be right for every environment and could cost more than the advantages may be worth.
Earlier this year at the Google Cloud Next event, Google announced the launch of its new managed service offering for multi-cloud environments, Google Cloud Anthos.
The benefits of public cloud, like cost savings and higher levels of productivity, are often presented as an “all or nothing” choice to enterprises. However, with this offering, Google is acknowledging that multi-cloud environments are the reality as organizations see the value of expanding their cloud platform portfolios. Anthos is Google’s answer to the challenges enterprises face when adopting cloud solutions alongside their on-prem environments. It aims to enable customers to evolve into a hybrid and multi-cloud environment to take advantage of scalability, flexibility, and global reach. In the theory of “write once, run anywhere”, Anthos also promises to give developers the ability to build once and run apps anywhere on their multi-cloud environments.
Anthos embraces open-source technology
Google Cloud Anthos is based on the Cloud Services Platform that Google introduced last year. Google’s vision is to integrate the family of cloud services together.
Anthos is generally available on both Google Cloud Platform (GCP) with Google Kubernetes Engine (GKE) and data centers with GKE On-Prem. So how does Google aim to deliver on the multi-cloud promise? It embraces open-source technology standards to let you build, manage and run modern hybrid applications on existing on-prem environments or in public cloud. Moreover, Anthos offers a flexible way to shift workloads from third-party clouds, such as Amazon Web Services (AWS) and Microsoft Azure to GCP and vice-versa. This allows users not to worry about getting locked-in to a provider.
As a 100% software solution, Anthos gives businesses operational consistency by running quickly on any existing hardware. Anthos leverages open APIs, giving developers the freedom to modernize. And, it automatically updates with the latest feature updates and security patches, because is based on GKE.
Rapid cloud transformation from Anthos
Google also introduced Migrate for Anthos at Cloud Next, which automates the process of migrating virtual machines (VM) to a container in GKE, regardless of whether the VM is set up on-prem or in the cloud lets users convert workloads directly into containers in GKE. Migrate for Anthos makes the workload portability less difficult both technically and in terms of developer skills when migrating.
Though most digital transformations are a mix of different strategies, for the workloads that will benefit the most, containers, migrating with Anthos will deliver a fast, smooth path to modernization according to Migrate for Anthos Beta.
Streamlining multi-cloud management with Anthos
Another piece of the offering is Anthos Config Management, which lets users streamline confirmation so they can create multi-cluster policies out of the box, set and enforce secure role-based access controls, resource quotas, and create namespaces. The capability to automate policy and security also works with their open-source independent service for microservices, Istio.
The management platform also lets users create common configurations for all administrative policies that apply to their Kubernetes clusters both on-prem and cloud. Users can define and enforce configurations globally, validate configurations with the built-in validator that reviews every line of code before it gets to the repository, and actively monitors them.
Expanded Services for Anthos
Google Cloud is expanding its Anthos platform with Anthos Service Mesh and Cloud Run for Anthos serverless capabilities, announced last week and currently in beta.
The first is Anthos Service Mesh, which is built on Istio APIs, is designed to connect, secure, monitor and manage microservice running in containerized environments, all through a single administrative dashboard that tracks the application’s traffic. This new service is aimed to improve the developer experience by making it easier to manage and troubleshoot the complexities of the multi-cloud environment.
Another update Google introduced was Cloud Run for Anthos. This managed service for serverless computing allows users to easily run stateless workloads on a fully managed Anthos environment without having to manage those cloud resources. It only charges for access when the application needs resources. Cloud Run for Anthos can run workloads on Google Cloud on on-premises and is limited to Google’s Cloud Platform (GCP) only.
Both AWS and Azure have hybrid cloud offerings but are not the same, mostly for one single reason.
AWS Outposts brings native AWS services, infrastructure, and operating models to virtually any data center, co-location space, or on-premises facility, in the same operating idea as Anthos, using the same AWS APIs, tools, and infrastructure across on-prem and the AWS cloud to deliver a seamless and consistent for an AWS hybrid experience.
As an extension of Azure to consistently build and run hybrid applications across their cloud and on-prem environments, Azure Stack delivers a solution for workloads wherever they reside and gives them access to connect to Azure Stack for cloud services.
As you can see, the main difference is that both AWS Outposts and Azure Stack are limited to combining on-premises infrastructure and the respective cloud provider itself, with no support for other cloud providers, unlike Anthos. Google Cloud Anthos manages hybrid multi-cloud environments, not just hybrid cloud environments, making it a unique offering for multi-cloud environment users.
Is green computing something cloud providers like Amazon, Microsoft, and Google care about? And whether they do or not – how much does it matter? As the data center market continues to grow, it’s making an impact not only on the economy but on the environment as well.
Public cloud offers enterprises more scalability and flexibility compared to their on-premise infrastructures. One benefit occasionally touted by the major cloud providers is that organizations will be more socially responsible when moving to the cloud by reducing their carbon footprint. But is this true?
Here is one example: Northern Virginia is the east coast’s capital of data centers, where “Data Center Alley” is located (and, as it happens, the ParkMyCloud offices), home to more than 100 data centers and more than 10 million square feet of data center space. Northern Virginia welcomed the data center market because of its positive economic impact. But as the demand for cloud services continues to grow, the expansion of data centers also increases dramatically. Earlier this year, the cloud boom in Northern Virginia alone was reaching over 4.5 gigawatts in commissioned energy, about the same power output needed from nine large (500-megawatt) coal power plants.
Environmental groups like Greenpeace have accused major cloud providers like Amazon Web Services (AWS) of not doing enough for the environment when operating data centers. According to them, the problem is that cloud providers rely on commissioned energy from energy companies that are only focused on dirty energy (coal and natural gas) and very little from initiatives that include renewable energy. While the claims bring the spotlight on energy companies as well, we wanted to know what (if anything) the major cloud providers are doing to rely less on these types of energy and provide data centers with cleaner energy to make green computing a reality.
Data Center Sustainability Projects from AWS
According to AWS’s sustainability team, they’re investing in green energy initiatives and are striving to commit to an ambitious goal of 100% use of renewable energy by 2040. They are doing this with the proposition and support of smart environmental policies, and leveraging expertise in technology that drives sustainable innovation by working with state and local environmental groups and through power purchase agreements (PPAs) from power companies.
AWS’s Environmental Layer, which is dedicated to site selection, construction, operations and the mitigation of environmental risks for data centers, also includes sustainability considerations when making such decisions. According to them, “When companies move to the AWS Cloud from on-premises infrastructure, they typically reduce carbon emissions by 88%.” This is because their data suggests companies generally use 77% fewer servers, 84% less power, and gain access to a 28% cleaner mix of energy – solar and wind power – compared to using on-premise infrastructure.
So, how much of this commitment has AWS been able to achieve and is it enough? In 2018, AWS said they had made a lot of progress in their sustainability commitment, and exceeded 50% of renewable energy use. Currently, AWS has nine renewable energy farms in the US, including six solar farms located in Virginia and three wind farms in North Carolina. AWS plans to add three more renewable energy projects, one more here in the US, one in Ireland and one in Sweden. Once completed they expect to create approximately 2.7 gigawatts of renewable energy annually.
Microsoft’s Environmental Initiatives for Data Centers
Microsoft has stated that they are committed to change and make a positive impact on the environment, by “leveraging technology to solve some of the world’s most urgent environmental issues.”
In 2016, they announced they would power their data centers with more renewable energy, and set a target goal of 50% renewable energy by the end of 2018. But according to them, they were able to achieve that goal by 2017, earlier than they expected. Looking ahead they plan to surpass their next milestone of 70% and hope to reach 100% of renewable energy by 2023. If they were to meet these targets, they would be far ahead of AWS.
Beyond renewable energy, Microsoft plans to use IoT, AI and blockchain technology to measure, monitor and streamline the reuse, resale, and recycling of data center assets. Additionally, Microsoft will implement new water replenishment initiatives that will utilize rainfall for non-drinking water applications in their facilities.
Google’s Focus for Efficient Data Centers
Google claims that making data centers run as efficiently as possible is a very big deal, and that reducing energy usage has been a major focus to them for over the past 10 years.
Google’s innovation in the data center market came from the process of building facilities from the ground up instead of buying existing infrastructures. According to Google, using machine learning technology to monitor and improve power-usage-effectiveness (PUE) and find new ways to save energy in their data centers gave them the ability to implement new cooling technologies and operational strategies that would reduce energy consumption in their buildings by 30%. Additionally, they deployed custom-designed, high-performance servers that use very little energy as possible by stripping them of unnecessary components, helping them reduce their footprint and add more load capacity.
By 2017, Google announced they were using 100% renewable energy through power purchase agreements (PPAs) from wind and solar farms and then reselling it back to the wholesale markets where data centers are located.
The Environmental Argument
Despite the pledges cloud providers are committing to in renewable energy, cloud services continue to grow beyond those commitments, and how much energy is needed to operate data centers is still very dependant on “dirty energy.”
Breakthroughs for cloud sustainability are taking place, whether big or small, providing the cloud with better infrastructures, high-performance servers, and the reduction of carbon emissions with more access to renewable energy resources like wind and solar power.
However, some may argue the time might be against us, but if cloud providers continue to better improve existing commitments that keep up with growth, then data centers – and ultimately the environment – will benefit from them.