Startup FInance

How Adobe Lost $80K a Day to Cloud Costs

In fall of 2018, the software company Adobe ran into a problem that would make any finance professional’s blood run cold: a development team at the company accidentally racked up $80,000 a day in unplanned charges for a computing job on Azure, the cloud computing service run by Microsoft. The mistake wasn’t discovered for more than a week. By the time it was, the bill had snowballed to well over half a million dollars.

Adobe isn’t the only prominent company to struggle with managing their cloud computing costs. Pinterest, Capital One, and Intuit have all been surprised by sudden increases in their bills from Amazon Web Services (AWS), according to reporting by popular tech site The Information. Adobe’s bill reportedly rose 64% from 2017 to 2018, while Pinterest’s increased 41% and Capital One’s spiked by 73%.

image.jpg

And it’s not just marquee brands, either. Cloud spending is increasing at organizations across the board. The median cloud spend for small-to-medium businesses is right around $120K — and 10% of SMBs are spending $1.2 million or more.

What that means is that for today’s CFOs, cloud management is no longer a “nice to have” — it’s mission critical. But as the team at Adobe can attest, getting a handle on cloud spending is easier said than done.

What does it take? Communication, collaboration, and tools that give stakeholders visibility into where spend is going and why. We’ll get to all those momentarily. But first, let’s take a look at how we got to this particular moment in computing history, and what’s at stake for finance leaders in mastering the cloud computing challenge.

It’s cloud computing’s world — we’re just living in it.

There’s no denying that cloud computing plays an increasingly central role in how business gets done in the 21st century. Cloud computing infrastructure and applications will total $214 billion in 2019 — up 17.5 percent from 2018, according to Gartner.

image.png

Increased cloud costs aren’t necessarily a bad thing. Sometimes, it’s a reflection of business growth. Others, it’s a strategic investment. Adobe, for example, began investing in additional cloud computing space in 2013, when they decided to switch their popular suite of products, including Photoshop and Illustrator, to a cloud-based subscription model. Intuit made a similar move with its core offerings, including TurboTax.

The switch from private servers to the cloud carries a number of advantages, including increased flexibility and agility. And at a time when computing power is no longer just a cost center, but a source of competitive market advantage, companies who don’t invest in cloud infrastructure could find themselves falling behind.

But there are challenges that come with the cloud, and monitoring spend is chief among them. According to one estimate, nearly 60% of organizations overspend their budget on cloud resources. And not all of that is money well-spent: A 2019 “State of the Cloud” report by Flexera finds that cloud users are wasting 35% of their spend. Doing some quick back-of-the-napkin math, if you’re an SMB spending roughly $120K on cloud services, that’s $42,000 in cloud spending going up in digital smoke.

And the stakes in getting cloud spending right are very real. In a survey of IT and finance professionals conducted by Cloudability, 80% of respondents said that poor cloud money management is a problem because it slows or halts cloud adoption (53%), cripples innovation (25%), lowers quality of service (38%), leads to sprawl and under-utilization of resources (40%), and increases cost (22%).

Screen Shot 2019-04-11 at 2.47.00 PM.png

All of this is happening at a time when spending within organizations is becoming increasingly decentralized. International Data Corporation (IDC) predicts that by 2020, more technology spend will be decided outside the IT department than within it.

CFOs and other financial leaders face a delicate balancing act when it comes to managing cloud costs. On the one hand, they seek to maintain good governance and ensure cost and risk are being managed effectively. On the other hand, they want to empower their team to get access to the tools and resources they need to do their jobs, and make investments that are going to move the business forward.

How does one find that balance? The successes and failures of leading companies provide insight into best practices in cloud spending and the steps financial leaders can take to be successful in this space.

Report Download:

Fyre Failure: What Billy McFarland Taught Us All About Startup Finance




How Capital One’s financially savvy CIO saved the company millions in server costs

When Capital One started migrating more of its computing tasks to AWS in 2015, the company’s Chief Information Officer, Rob Alexander, issued a mandate to his team: get your applications up and running on Amazon’s computers as quickly as possible. Capital One’s private data center leases were expiring. If the team missed the deadline, the company could be on the hook for a lengthy and expensive new contract.

Once the migration was complete, Alexander issued a new directive: get those applications running as efficiently as possible. According to a former Capital One employee, that included fine-tuning the company’s cloud server configurations, and shutting down application-testing servers at night and on weekends.

In other words: Alexander’s leadership as his team transitioned into the cloud shows that he had a clear view of what was at stake for Capital One in the move, both technically and financially.

IT is the first line of defense when it comes to managing cloud spend. According to the Cloudability survey, 80% of IT respondents said they are notified of potential overspend before it takes place, while 68% of finance respondents said they are alerted only after it’s too late.

It’s worth noting here how quickly cloud costs like these can add up if you don’t monitor them carefully. Take the nights and weekends example: according to an estimate by ParkMyCloud, about 44% of cloud spend is on non-production resources like the ones Alexander directed his team to shut down. Most non-production resources are only used during the workweek, leaving 128 hours — fully 76% of the week — when these resources are not actively in use. For our median SMB cloud spend of $120K, that translates into:

image.png

in unnecessary cloud spending if those non-production cloud resources are left running during non-work hours.

For a company like Capital One, which spent over $250 million on AWS services in 2018, the math looks like this:

Screen Shot 2019-04-15 at 11.16.46 AM.png

in wasted cloud spend if non-production cloud resources are left running.

Despite IT’s crucial role in heading off potential cloud waste, effective communication between IT and finance remains rare. Only 28% of professionals say IT and finance leaders within their organization communicate about cloud spending in a formal reporting capacity.

Part of the problem is that there are conflicting priorities between the two departments. A recent Forrester study sponsored by SAP Concur found that 61% of firms believe IT is focused more on usability and employee experience and less on spending reductions, while 64% of companies say that finance is focused more on reducing spending and less on usability and employee experience.

This can be solved, at least in part, by improved communication between an organization’s tech and finance leader. Regular touch points can help ensure alignment on strategy and promote understanding of each team’s perspective. The more the organization is growing—and the faster the situation on the ground changes—the more frequent those touch points should be. Centralizing financial conversations in a single location where stakeholders across departments can see and participate in them, such as a dedicated Slack channel, can also help facilitate the flow of information between teams.

The days when IT and finance could plug away in their individual silos are gone. As cloud computing continues to shift to the center of business operations, organizations that master finance-to-IT communication will have a distinct advantage.

How an unforeseen traffic spike cost Pinterest $20 million

Another prominent organization that encountered unexpected cloud costs is the popular social media site Pinterest. The company spent approximately $190 million on AWS services in 2018—$20 million more than they had originally budgeted.

One culprit in the overspend was an unforeseen spike in traffic around the holiday season. As users visited Pinterest to get decoration inspiration, look up festive recipes, and plan their holiday shopping, the surge in traffic led to a spike in Pinterest’s cloud utilization. The company had paid in advance for their AWS services for the year, which meant that when the extra traffic hit, they had to buy additional capacity at a significantly higher price.

The most painful part of Pinterest’s story is that the unexpected charges could have been avoided if the company had anticipated the holiday spike in traffic. This highlights the importance of planning usage in advance and setting limits and restrictions appropriately.

All of the major cloud providers, including Microsoft Azure, AWS, and Google, provide tools aimed at helping customers manage cloud costs. The exact suite of features varies from provider to provider, but they typically include tools like a pricing calculator, billing APIs, and automated notifications of spending anomalies and overspending risks.

And while paying for cloud computing capacity upfront was ultimately Pinterest’s undoing, that doesn’t mean upfront purchases are a bad idea. Cloud service providers usually offer heavily discounted pricing if customers pay for computing capacity in advance. It’s a valuable cost-saving measure, as long as companies accurately project their needs.
Screen Shot 2019-04-12 at 4.56.07 PM.png

A breakdown of cloud cost centers by category (Source: Rightscale)

It is recommended to plan cloud capacity requirements at least two quarters to one full year in advance, keeping the following in mind:

  1. Ensure full understanding of the provider’s pricing model.
  2. Account for all cloud computing services, including data, storage, and networking costs as well as computing infrastructure.
  3. Incorporate foreseeable fluctuations in cost into planning.

How streamlining ownership helped Intuit control cloud spend

Intuit is another Fortune 500 company that has seen its cloud bills skyrocket in recent years. The financial services organization saw its AWS bill rise 93% to around $145 million in 2018, in large part due to a decision to migrate TurboTax and other Intuit products to the cloud.

As Intuit’s cloud investment has grown, the company has taken measures to ensure that investment is being spent responsibly. According to VP of Product Infrastructure Sembian Krishnamurthy, the company became more stringent about who has access to AWS and required AWS accounts to be tied to individual employees and departments to make tracking easier.

As corporate purchasing in general is becoming more decentralized, cloud spending decisions are increasingly being made at the point of the end-user: the individual programmers using the space in question. While this model has some benefits— team members are able to get the server capacity they need, when they need it—it can create a “too many cooks in the kitchen” effect, with dozens or even hundreds of individuals making decisions about cloud service purchases across the organization.

By adopting tools that track every cloud purchase to the individual who made it, companies can maintain tight control on their cloud spending. Whether that be customized workflows that ladder up to a smaller number of key decision makers, or virtual cards that make it easy to trace each transaction, having visibility into cloud spend can help organizations stay within budget.

Effective cloud cost management is all about finding the balance between empowerment and visibility, flexibility and control. With the right tools and practices in place, CFOs can strike that balance for their organizations.

Head in the cloud, feet on the ground

The job of a CFO is not simply to reduce costs; it’s to ensure costs are being managed effectively and resources are being channeled to yield company growth. As IT spending continues to shift, cloud costs will play an increasingly central role in the process.

By centralizing purchase approvals, planning usage strategically, and improving communication between stakeholders about resource allocation, CFOs can steer their organization clear of the kind of over-spending catastrophe that hit Adobe, while still facilitating innovation that will move the company forward.