Why Google’s Aiming Multi-Cloud with BigQuery Omni

Why Google’s Aiming Multi-Cloud with BigQuery Omni

During its virtual Google Cloud Next ’20 “On Air” series, Google announced the introduction of BigQuery Omni. This is an extension of its existing BigQuery data analytics solution to now analyze data in multiple public clouds, currently including Google Cloud and Amazon Web Services (AWS), with Microsoft Azure coming soon. Powered by Google Cloud’s Anthos, and using a unified interface, BigQuery Omni allows developers to analyze data locally without having to move data sets between the platforms.

BigQuery Engine to Analyze Multi-Cloud Data

Google Cloud’s general manager and VP of engineering, Debanjan Saha, says “BigQuery Omni is an extension of Google Cloud’s continued innovation and commitment to multi-cloud that brings the best analytics and data warehouse technology, no matter where the data is stored.” And that, “BigQuery Omni represents a new way of analyzing data stored in multiple public clouds, which is made possible by BigQuery’s separation of compute and storage.” 

According to Google Cloud, this provides scalable storage that can reside in Google Cloud or other public clouds, and stateless, resilient compute that executes standard SQL queries. 

Google Cloud reports that BigQuery Omni will:

  • Break down silos and gain insights on data with a flexible, multi-cloud analytics solution that doesn’t require moving or copying data from other public clouds into Google Cloud for analysis. 
  • Get consistent data experience across clouds and datasets with a unified analytics experience across datasets, in Google Cloud, AWS, and Azure (coming soon) using standard SQL and BigQuery’s familiar interface. BigQuery Omni supports Avro, CSV, JSON, ORC, and Parquet.
  • Securely run analytics to another public cloud with a fully managed infrastructure, powered by Anthos, so you can query data without worrying about the underlying infrastructure. Users can choose the public cloud region where their data is located, and run the query.

Why is Google Aiming Multi-Cloud?

Many organizations leveraging public cloud are doing so with multiple clouds: 55% of organizations are multi-cloud according to a recent survey from IDG, and 80% according to a recent Gartner survey. (Is this actually necessary? Maybe.)

Google Cloud has been the most open to supporting this multi-cloud reality, and perhaps implicit in releases like Anthos and BigQuery Omni is Google’s recognition that it’s #3 in the market, and many of its customers have a presence in AWS or Azure.

So, BigQuery Omni actually involves physically running BigQuery clusters in the cloud on which the remote data resides. This is something that in the past, could only be done if your data was stored only in Google Cloud. Now with Kubernetes-powered Anthos, as well as the visualization tool gained in Google’s acquisition of Looker, Google is moving toward a middleware strategy. Now, it is offering services to bridge data silos, as a strategy to gain market share from its bigger competitors. Expect to see more similar service offerings coming from Google as they look to break AWS’s lead on public cloud.

How to Use Azure Spot Virtual Machines for Cost Savings

How to Use Azure Spot Virtual Machines for Cost Savings

Azure Spot Virtual Machines are a purchasing option that can save significant amounts on infrastructure, for certain types of workloads.  Azure Spot VMs are not created as a separate type of VM in Azure, instead, it’s a capability to bid for spare capacity at a discount from on demand pricing. But there’s one caveat: at any given point in time when Azure needs the capacity back, the Azure infrastructure will deallocate and evict the Spot VM from your environment.

In the past, Azure offered Low Priority VMs, which were charged at a fixed price. In March this year, that option was replaced by Azure Spot VMs. With the newer option, you bid by indicating the maximum price you are willing to pay for a VM.

Why Use Azure Spot VMs

Microsoft allows you to use their unused compute capacity at a discounted rate. These discounts are variable and can go up to >90% of the pay-as-you-go rates, depending on the size of the VM and the unused capacity available. The amount of capacity available can vary based on region, time of day, and more. 

You can use Azure Spot VMs for workloads that are not critical or need to run 24×7. For example, a basic scenario would be for testing the load of a particular workload that you want to perform for a fraction of the cost. Other use cases include batch processing, stateless applications that can scale out, short-lived jobs that can be run again if the workload is evicted, etc.

Keeping in mind that there are no SLAs or availability guarantees for these Spot VMs. The most significant concern users have is that they may not be available to you to get resources, especially at peak load times. The issue is not with the service, it’s with how it is intended to work. Be aware of this when making the decision to use this approach.

Some important things to consider when using Azure Spot VMs: 

  • VMs are evicted based on capacity or by if the price exceeds your maximum set price 
  • Azure’s infrastructure will evict Spot VMs if Azure needs the capacity for pay-as-you-go workloads
  • B-series and promo versions of any size (like Dv2, NV, NC, H promo sizes) are not supported
  • A Spot VM cannot be converted to a regular VM or vice versa. You would have to delete the VM and attach the disk to a new VM
  • VMs that are evicted and deallocated are not turned back on when capacity or price comes back inside allowed limits, you will need to manually turn them back on
  • You will be unable to create your VM if the capacity or pricing are not inside the allowed limits

How to Use Azure Spot VMs

You have two choices when deploying Azure Spot VMs. When you enable the feature in your Azure environment, you need to select what type of eviction and eviction policy you want for the capacity: 

Types of eviction:

  • By capacity only – the VM is evicted when Azure needs capacity. In other words, your maximum price for the spot VM is the current price of the regular VM
  • By maximum price – the VM is evicted when the spot price is greater than the maximum price

Eviction policy (currently available): 

  • Stop / Deallocate

The eviction policy for Spot VMs is set to Stop / Deallocate which moves your evicted VMs to the stopped-deallocated state, allowing you to redeploy the evicted VMs at a later time. Remember reallocating Spot VMs will be dependent on there being available Spot capacity. However, the deallocated VMs will count against your spot vCPU quota and you will be charged for your underlying disks. If your Spot VM is evicted, but you still need capacity right away, Azure recommends you use a standard VM instead of Spot VM. 

Do Azure Spot VMs Save You Money?

Yes: these discounted VMs can save you money. How much will vary? Azure Spot VMs prices are not fixed like standard instances, they change over the day and vary based on the supply and demand in a particular region. 

Azure Spot VMs are a good option that can provide cost savings if your application can handle unexpected interruptions. 

Use Spot VMs as part of your full cost-saving strategy. For on-demand workloads that aren’t needed 24×7, ensure you have appropriate on/off schedules in place. All VMs should be properly sized to the workload. You can start automating these Azure cost optimization tasks with ParkMyCloud today.

Google Sustainability Efforts in the Cloud Now Claim to be “Carbon Intelligent”

Google Sustainability Efforts in the Cloud Now Claim to be “Carbon Intelligent”

Google Sustainability is an effort that ranges across their business, from the Global Fishing Watch to environmental consciousness in the supply chain. Given that cloud computing has been a major draw of global energy in recent years, the amount of computing done in data centers more than quintupled between 2010 and 2018. But, the amount of energy consumed by the world’s data centers grew only six percent during that period, thanks to improvements in energy efficiency. However, that’s still a lot of power. That’s why Google’s sustainability efforts for data centers and cloud computing are especially important. 

Google Cloud Sustainability Efforts – As Old as Their Data Centers

Reducing energy usage has been an initiative for Google for more than 10 years. Google has been carbon neutral since 2007, and 2019 marked the third year in a row that they’ve matched their energy usage with 100 percent renewable energy purchases. Google’s innovation in the data center market also comes from the process of building facilities from the ground up instead of buying existing infrastructures and using machine learning technology to monitor and improve power-usage-effectiveness (PUE) and find new ways to save energy in their data centers. 

When comparing the big three cloud providers in terms of sustainability efforts, AWS is by far the largest source of carbon emissions from the cloud globally, due to its dominance. However, AWS’s sustainability team is investing in green energy initiatives and is striving to commit to an ambitious goal of 100% use of renewable energy by 2040 to become as carbon-neutral as Google has been. Microsoft Azure, on the other hand, has run on 100 percent renewable energy since 2014 but would be considered a low-carbon electricity consumer and that’s in part because it runs less of the world than Amazon or Google. 

Nonetheless, data centers from the big three cloud providers, wherever they are, all run on electricity. How the electricity is generated is the important factor in whether they are more or less favorable for the environment. For Google, reaching 100% renewable energy purchasing on a global and annual basis was just the beginning. In addition to continuing their aggressive move forward with renewable energy technologies like wind and solar, they wanted to achieve the much more challenging long-term goal of powering operations on a region-specific, 24-7 basis with clean, zero-carbon energy.

Why Renewable Energy Needs to Be the Norm for Cloud Computing

It’s no secret that cloud computing is a drain of resources, roughly three percent of all electricity generated on the planet. That’s why it’s important for Google and other cloud providers to be part of the solution to solving global climate change. Renewable energy is an important element, as is matching the energy use from operations and by helping to create pathways for others to purchase clean energy. However, it’s not just about fighting climate change. Purchasing energy from renewable resources also makes good business sense, for two key reasons:

  • Renewables are cost-effective – The cost to produce renewable energy technologies like wind and solar had come down precipitously in recent years. By 2016, the levelized cost of wind had come down 60% and the levelized cost of solar had come down 80%. In fact, in some areas, renewable energy is the cheapest form of energy available on the grid. Reducing the cost to run servers reduces the cost for public cloud customers – and we’re in favor of anything that does that.
  • Renewable energy inputs like wind and sunlight are essentially free – Having no fuel input for most renewables allows Google to eliminate exposure to fuel-price volatility and especially helpful when managing a global portfolio of operations in a wide variety of markets.

Google Sustainability in the Cloud Goes “Carbon Intelligent”

In continuum with their goals for data centers to consume more energy from renewable resources, Google recently revealed in their latest announcement that it will also be time-shifting workloads to take advantage of these resources and make data centers run harder when the sun shines and the wind blows. 

“We designed and deployed this first-of-its-kind system for our hyperscale (meaning very large) data centers to shift the timing of many compute tasks to when low-carbon power sources, like wind and solar, are most plentiful.”, Google announced.  

Google’s latest advancement in sustainability is a newly developed carbon-intelligent computing platform that seems to work by using two forecasts – one indicating future carbon intensity of the local electrical grid near its data center and another of its own capacity requirements – and using that data “align compute tasks with times of low-carbon electricity supply.” The result is that workloads run when Google believes it can do so while generating the lowest-possible CO2 emissions.

The carbon-intelligent computing platform’s first version will focus on shifting tasks to different times of the day, within the same data center. But, Google already has plans to expand its capability, in addition to shifting time, it will also move flexible compute tasks between different data centers so that more work is completed when and where doing so is more environmentally friendly. As the platform continues to generate data, Google will document its research and share it with other organizations in hopes they can also develop similar tools and follow suit. 

Leveraging forecasting with artificial intelligence and machine learning is the next best thing and Google is utilizing this powerful combination in their platform to anticipate workloads and improve the overall health and performance of their data center to be more efficient. Combined with efforts to use cloud resources efficiently by only running VMs when needed, and not oversizing, resource utilization can be improved to reduce your carbon footprint and save money.

NEW in ParkMyCloud: Now Offering Azure AKS Cost Optimization!

NEW in ParkMyCloud: Now Offering Azure AKS Cost Optimization!

We’re excited to share the latest in cost optimization for container services: ParkMyCloud now enables enterprises to optimize their Azure AKS (managed Azure Kubernetes Service) cloud costs. This is the second managed container service supported in the platform since we announced support for the scheduling of Amazon EKS (managed Elastic Kubernetes Service) last month.

Why is Container Cost Optimization Essential?

As we continue to expand our container management offering, it’s essential to understand that container management, like the broader cloud management, includes orchestration, security, monitoring, and of course, optimization.

Containers provide opportunities for efficiency and more lightweight application development, but like any on-demand computing resource, they also leave the door open for wasted spend. If not managed properly unused, idle, and otherwise suboptimal container options will contribute billions more to the estimated $17.6 billion in wasted cloud spend expected this year alone.

AKS Scheduling in ParkMyCloud

The opportunities to save money through container optimization are in essence no different than for your non-containerized resources. ParkMyCloud analyzes resource utilization history and creates recommended schedules for compute, database and container resources, and programmatically schedules and resizes them, saving enterprises around the world tens of millions of dollars.

You can reduce your AKS costs by setting schedules for AKS nodes based on working hours and usage, and automatically assign those schedules using the platform’s policy engine and tags. Or, use ParkMyCloud’s schedule recommendations for your resources based on your utilization data. 

Already a ParkMyCloud user? Log in to your account to optimize your AKS costs. Please note you’ll have to update your Azure permissions. Details available in the release notes

Not yet a ParkMyCloud user?start a free trial to get started.

What’s Next for Container Optimization?

This is the second release for container optimization in ParkMyCloud. The platform already offers support for Amazon EKS (managed Elastic Kubernetes Service). Support scheduling for Amazon ECS, AWS Fargate, and Google Kubernetes Engine (GKE) will be available soon in the next few months, so stay tuned.

Questions? Feature requests? We’d love to hear them. Comment below or contact us directly.

Further reading:

 

Will Robotic Process Automation Save Your Company Time and Money in the Cloud?

Will Robotic Process Automation Save Your Company Time and Money in the Cloud?

After hearing a lot of buzz about this concept in AI, we decided to see what’s next for robotic process automation. The promise of the technology is that it can automate processes that employees are doing manually, saving your employees’ time and potentially reducing operational costs. While robotic process automation (RPA) interest has been high for a while, actual adoption is now catching up and will only continue to grow in the future. Organizations are understanding the power of process automation, so in turn, more industries are expected to deploy more RPA bots to eliminate manual repetitive actions performed by employees. 

RPA software is en route to becoming a billion-dollar category in 2020. Last year, Gartner projected that spending on RPA software was expected to hit $1.3 billion. However, there are still some growing pains to address with RPA and is not exactly a 100 percent perfect, but it fits right in with the current trends in cloud computing toward optimization. And, since, we’re all about saving time and money – let’s recap on this trend to see how it can help to do these things. 

What is Robotic Process Automation?

To recount, RPA, whether it’s called “intelligent automation” or “cognitive automation” in the future, is a way to automate business processes by creating software robots paired with artificial intelligence (AI) and machine learning capabilities to perform manual and mundane work-tasks. It allows users to configure within an application and gives them the capability to handle a variety of repetitive tasks by processing, employing, generating and communicating information automatically. For example, you might program RPA bots to do first-level customer support tasks by searching for answers; copy and paste data from one system to another for invoicing or expense management or issue refunds. This video from IBM shows an example in action. 

RPA software is not part of an organization’s IT infrastructure. Instead, it sits on top of it, enabling a company to implement the technology quickly and efficiently. Furthermore, RPA tools can be trained to make judgments about future outputs. Many users appreciate its non-intrusive nature and the ability to integrate within infrastructures without causing disruption to systems already in place.

How can you use Robotic Process Automation?

RPA technology can help organizations on their digital transformation journeys by:

  • Enabling better customer service.
  • Ensuring business operations and processes comply with regulations and standards.
  • Allowing processes to be completed much more rapidly.
  • Providing improved efficiency by digitizing and auditing process data.
  • Creating cost savings for manual and repetitive tasks.
  • Enabling employees to be more productive.

Companies like Walmart, AT&T, and Walgreens are adopting the use of RPA. Clay Johnson, the CIO of Walmart, says they use RPA bots to automate pretty much anything from answering employee questions to retrieving useful information from audit documents. The CIO of American Express Global Business Travel, David Thompson, says they implement the use of RPA to automate the process for canceling an airline ticket and issuing refunds. In addition, Thompson is looking to use RPA to facilitate automatic rebooking recommendations, and to automate certain expense management tasks in the company.

But more specific to cloud computing and IT, one great application for RPA is in automated software testing. If testing involves multiple applications and monotonous work, RPA can replace workers’ time spent testing. Automated tests can run repeatedly at any time of day. This approach fits in with continuous testing as well as continuous integration (CI) and continuous delivery (CD) software development practices. Additionally, RPA can be used to automate processes in monolithic legacy systems that are not worth developers’ time to update, to bring automation while work on newer microservices systems is in progress. 

Is Robotic Process Automation the Best Way to Automate Cost Control?

A study found that not all automation is achievable with RPA. In the study, they conclude that only three percent of organizations have managed to scale RPA to a high level. Additionally, Gartner placed RPA tools at the “Peak of Inflated Expectations” in their Hype Cycle guide for artificial intelligence  – another vote for more buzz than potential. In reality, it is only as efficient as the person configuring the automation flow and organizations that have overly idealized expectations of the technology’s capabilities. Those that don’t have a solid grasp of their own processes may find it difficult to find the right tool to automate jobs.

However, RPA is expected to deliver tangible results to organizations that make automation a key component of their digital transformation as the collaboration between digital workers and human talent become more efficiently aligned in the future.

So can it save you time and money? If employees at your company are spending a large percentage of their time on repetitive tasks that require little to no decision making, then yes, it probably can. It’s also important because it will free up developer time that is spent on automatable tasks, like scripting, so they can focus on creating value for your business. 

For complex and long-term automation, though, purpose-built software is a better solution. If there is already a solution to your automation needs on the market, it will probably serve you better than RPA because there won’t be an upfront period needed to program bots, you won’t need to make frequent changes to your processes like many RPA bots will require, and it’s a better solution for the long run.