The extreme complexity of public cloud pricing and billing makes the correct selection of cost management tools vital for any organization. There are three primary toolsets to choose from:
- The cloud vendors’ own web-based billing consoles
- Relational databases, which are used to house and query billing files
- Commercially offered SAAS cloud cost management platforms
Let’s examine the use cases and pros and cons of each of these toolsets.
Cloud vendor billing consoles
The public clouds all offer web-based billing consoles with basic levels of historical cost reporting, budgeting, forecasting, and alerting. They may also offer recommendations for “rightsizing” cloud resources to cut costs and/or recommendations for participation in vendor discount programs such as reserved instances, committed use discounts, etc. These consoles generally require minimal configuration, are provided free as part of the cloud services, and usually filter the cloud billing data to the consumption of the user who has logged on to the console. When users’ needs are limited to fairly basic reporting, these consoles work fine. However, FinOps practitioners and technical teams may have needs that exceed the consoles’ capabilities, such as:
- The ability to enrich the cloud billing data with other datasets
- The ability to easily customize the subset of accounts against which reports run
- The ability to report on multiple clouds’ spend in a single system
- The ability to generate a wider range of customized reports
- The ability to generate reports with resource ID-level granularity (i.e. detailing specific compute instances, storage volumes, etc.)
- The need for enhanced or expanded optimization recommendations
The specific capabilities of the vendors’ billing consoles varies. Whether an organization should begin supplementing the standard console with additional tools will be decided based on their needs, the cloud(s) they are consuming, and their internal data science capabilities.
Relational databases
Use of relational databases to analyze cloud spend provides excellent reporting capabilities, but is also the most challenging toolset to set up and administer. The database needs to be configured, billing files supplied by the cloud vendor must be loaded into the database, meaningful queries for the desired reporting must be crafted, and a Business Intelligence (BI) tool needs to be connected to the database tables. Depending on the cloud vendor being used, the difficulty level of ingesting the cloud billing files into a relational database ranges from “trivially easy” to “extremely complex.” In the case of Google Cloud Platform (GCP), the process is easy: one simply ticks a box in the billing console and specifies a destination dataset and table in Google’s BigQuery database service. Cloud cost data for GCP begins flowing in immediately and is ready to be queried directly in the BigQuery web client.
The process for other clouds can be more complicated. Amazon Web Services (AWS) provides an integration between their billing file, the Cost And Usage Report (“CUR”), and their Redshift database service, but this “integration” is limited to the creation of an SQL script that must be edited for authentication and other parameters before it can be executed. Redshift must access the “S3” object storage “bucket” (or logical folder) where the CUR is stored, and depending on the scenario, the process of configuring permissions between Redshift and S3 can be extremely complex. A number of fairly advanced skills must therefore be employed to import the CUR into Redshift, spanning security, Redshift cluster administration, and SQL client configuration.
Once all of that work is done, however, almost limitless reporting options become possible. Relational databases provide the ability to report at an extremely granular level and to enrich the billing data with external datasets that will provide the organization new insights into their spend. An over-simplified example of enriching billing data with external datasets can illustrate the advantages.
In many organizations, the cloud billing data may not contain all the data needed to “close the books” each month (to report cost per business unit, for example). This data shortfall is seldom the cloud vendor’s “fault” per se, but is rather a function of the necessary data being unrelated to and independent of the rest of the cloud billing data. For example, consider a scenario wherein an enterprise wishes to be able to report costs by financial and technical “responsible parties” (or “RPs”) – i.e. team members who are accountable for the finance and technical aspects of the cloud computing being used in their department. This graphic shows an over-simplified example of the type of “account table” or “account mapping” many organizations maintain in spreadsheets to link individuals to business units:
It would not be possible to report on the spend each of these RPs is incurring in the console unless the relationship between RP and business unit was reflected in tagging or labeling. The example above has only two fields linked to business units, but in the real world these account tables often contain too many additional fields (such as accounting charge codes, regions, etc.) for tagging to be implemented for all of them. In these situations, reporting is often done “out of band” by exporting a console report and using a spreadsheet to look up the appropriate values. Apart from the effort needed to generate such a report, the spreadsheet will need to be distributed manually to everyone who wishes to consume the report–another cumbersome process that will limit the visibility the organization has into their costs. Enriching the billing data with a relational database can solve this problem. The account table can be uploaded to a relational database along with the cloud cost data, and queries can be employed to enrich the billing data with the appropriate RP names (and other account information such as general ledger codes, etc). With those queries run, it is very easy to iterate through a variety of reports against the data, such as this report showing costs by RP:
Better still, with all of the data in a central location, any number of users can generate such reports using any number of different Business Intelligence tools such as Google Data Studio, Microsoft PowerBI, Tableau, etc. This broad access to fully customizable reporting greatly increases the organization’s visibility into their spend, which is the first and most critical step of operating efficiently in the cloud. The “account table” example explored here is only one of an almost limitless range of external datasets that can prove useful. In other cases, stand-alone queries can enrich the billing data to give new insights without the need to bring in external data. At SADA, for example, we provide our customers with a library of queries that greatly improves their ability to identify opportunities for cost savings by enriching the billing data with custom dimensions and metrics. Importing the vendor billing file into a relational database also provides an incredibly granular view of the enterprise’s cloud economics, including the ability to report down to the resource ID level.
The best candidates for use of relational databases will be those organizations with both a significant need for expanded reporting capabilities and the right internal capabilities to set up and operate the database tooling.
Commercial SAAS platforms
In many cases, commercially available cost management platforms such as Ternary, Cloudability, and Cloudhealth provide an ideal middle ground between the simplicity of the billing console and the complexity of a relational database. Initial configuration of these tools is fairly simple and is primarily handled by the vendor.
Most offerings allow costs for multiple clouds to be managed in a single platform, which is a huge advantage. Because they have their own means of providing authentication (often via single sign on (SSO)), these platforms offer increased flexibility when creating customizable “Views” to isolate reporting to a sub-group within an organization such as a business unit or department. Kyle McLaughlin, Director of Global Cloud Advisory Services at Apptio, says this solves a major problem for many cloud consumers.
According to McLaughlin, “The customers that implement FinOps the most effectively have highly federated account structures with, for example, separate development and production accounts and sub accounts within those. If you are an engineer who works on three different applications, you would need to jump around between 9-10 different accounts to manage your costs via the cloud’s console, potentially encountering complex security challenges in the process. In Cloudability you can report against those applications in minutes on a single pane of glass by simply applying filters.”
The advantage is even greater when one considers the fact that many finance and accounting professionals with a need to access cloud cost data may not initially have logins to the cloud vendor consoles at all.
Cloud cost management platforms also offer the ability to enrich cloud cost data with external data, albeit in a more limited way than relational databases do. Ternary, for example, offers a very innovative set of collaboration tools that allows technical teams to track which team members have reviewed and are executing Ternary’s cost optimization recommendations.
Patrick Raymond, co-founder and Chief Product Officer at Ternary, says, “In order to operate FinOps at scale, you need visibility into your spend but you also need to be able to have a conversation about things. In the cloud vendor’s console, you get a list of recommendations but you have no way to track your progress on executing them. Using Ternary, you can track who is working on specific recommendations, what has been implemented, and how much you have saved.”
Cloudability’s “Business Dimensions” and “Business Metrics” features also allow external data to enrich the cloud billing data. McLaughlin relates a success story wherein the Business Metric feature gave a Cloudability customer visibility into the unit costs of their services, resulting in substantial savings.
“We have a video streaming customer that measures most of their business outcomes based on video plays. Cloudability’s Business Metrics allowed them to enrich their billing data with video play data from all of their platforms. This in turn allowed them to measure their production costs per video play and to measure the business value of individual new features they were considering. With this new visibility into unit costs, they reduced the cost of one new feature by 80 percent,” says McLaughlin.
These commercial offerings also offer significantly expanded reporting capabilities when compared to most vendor consoles. Raymond says that Ternary’s reporting capabilities rival those that can be provided by relational databases without the need for complex configuration and data science skills.
“To generate reports similar to Ternary’s with a relational database, you would need skilled engineers who are familiar with the data structure to export the data and generate queries. Ternary allows anyone who is involved with the budget the ability to write complex SQL queries by literally dragging and dropping in the interface,” Raymond says.
Consumers of these cloud cost management platforms can also benefit from the vendors’ professional expertise in cloud financial management. Marie Burke, Director of Product Marketing for VMware’s CloudHealth platform, says, “Helping customers understand what tooling and reporting features will be most impactful for a particular client is key, but we don’t want to focus only on the technology. Cost management is about the people and processes as well. For example we had one customer that worked directly with our Customer Success team over several engagements. They created a quarterly business review process modeled after the one we use with our customers that allowed them to drive continuous best practices across their organization. Our goal is to encourage our clients to go beyond individual tactics and focus on enhancing their expertise.”
Cloud cost management vendors like CloudHealth offer paid professional services to augment their platforms in engagements such as the one Burke references, but even in the absence of formal consulting engagements, users of these platforms will benefit from the FinOps expertise of the vendors’ support and customer success teams without incurring additional cost.
As paid commercial offerings, the cost of these platforms will naturally need to be a consideration as to whether they are right for any particular organization. Our experience has shown us that for most organizations, the ROI on these tools is usually positive and in some cases can actually recover multiples of the platform’s costs over time. Some customers argue that they have the internal capabilities to replicate many of the features of these tools, but our advice is to direct those talents toward the business’s core competencies as opposed to re-creating technologies that already exist and are being maintained by a third party.
For most organizations, therefore, we recommend that vendor consoles be augmented with a commercial cost management platform at a minimum. Assuming they have sufficient capabilities to operate a relational database, we have found that almost all organizations will benefit from combining all three cost management toolsets to maximize visibility into their spend.
