AccuWeather enhances data integration and business intelligence services in the cloud

When you’re searching for a hyperlocal, seven-day weather forecast while planning a barbecue or receiving real-time severe weather alerts on your digital devices, there’s a good chance that AccuWeather, the most accurate and most used source of weather forecasts and warnings, is behind the weather information you receive.

AccuWeather, which studies indicate provides the most accurate weather forecasts and warnings, has saved tens of thousands of lives, and prevented hundreds of thousands of injuries and tens of billions of dollars in property damage since their founding in 1962. The US-based company is a global organization forecasting for every longitude and latitude point on earth and responding to more than 45 billion data requests daily from everywhere in the world.

On a daily basis, AccuWeather impressively delivers weather forecasts, warnings, and insights to billions of people to save lives and protect people and property through digital media properties, such as its accuweather.com website and free mobile application, as well as radio, television, newspapers, digital signage, the 24/7 AccuWeather Network cable channel and the AccuWeather NOW streaming service.

Business challenge 

Beyond the considerable amount of weather data, such as temperatures, humidity, the largest collection of weather data on the planet and more forecasting models than any other weather source, AccuWeather also processes a vast amount of CRM and sales data for analysis to make decisions and execute the company’s performance marketing business.

“On my side of the fence, with digital analytics and data, it all comes down to speed to insights,” says Asif Rahman, Vice President of Digital Analytics and Audience at AccuWeather. “Optimizing the process to query and access data in the most efficient manner possible is the key to our success.”

This proved daunting when the analytics team attempted to obtain, process, or push audience data from six different data sources, including Google Analytics, Pushly, Airship, and their iOS and Android apps. Each source of incoming data was in a variety of formats, and a few did not have API endpoints, resulting in a very archaic and chaotic system that sometimes failed (where the workaround involved receiving emailed CSV files and running a script to upload data in the system).

“The analytics team found themselves diverting their energy in wrangling data as opposed to doing what they do best, which is providing business intelligence,” says Rahman. “My biggest challenge was the need to have all of this in an easy to access, centralized location. Specifically, we need to get data from point A to point B and do some ‘ETL-ing’ in the middle.” ETL is a three-phase process where data is extracted, transformed, and loaded into an output data container.

Solution

AccuWeather’s front-end digital operations already utilized many Google solutions, such as Google Analytics to track and report on 15 billion events each month, Google’s ad serving platform, DoubleClick for Publishers, and the BigQuery managed data warehouse service.

Upon a recommendation from Google Cloud Support, AccuWeather partnered with SADA, a multiple-time Google Cloud Sales Partner of the Year, to build a centralized data warehouse and develop a proper ETL process to ingest and transform their data.

“We needed to slice data into smaller, meaningful data sets that we could query every single day without running everything and the kitchen sink,” says Rahman. “So we looked at optimization in terms of how much data we can access, optimization in terms of what it costs to access that data, and then actually optimizing data in a formal pipeline.”

SADA exposed AccuWeather to a number of useful technologies in setting up their formalized ETL pipeline.

SADA opened our eyes to some Google Cloud products that we weren’t leveraging, like Cloud Run, Cloud Build, and Cloud Composer. We were aware of Pub/Sub functions, but we lacked the hands-on expertise to take full advantage of such solutions. The same goes with some unique capabilities and features of BigQuery.

Asif Rahman | Vice President of Digital Analytics and Audience at AccuWeather

In the end, SADA engineers architected and wrote code for AccuWeather’s customized ETL pipeline, which successfully mapped all the incoming data from six sources into a single warehouse. This enabled AccuWeather to conduct analytics in a seamless and automated manner.

Results

As a result of working with SADA, AccuWeather was able to modernize their data integration and business intelligence services. From an analytics system that often required batch uploads, AccuWeather has transformed their data analysis into a smoother functioning, optimized process. With BigQuery at the center, data efforts have been cut by almost one-third. Overall, SADA helped AccuWeather: 

• Establish an ETL data pipeline powered by BigQuery
• Follow best practices to grow and expand data availability
• Leverage Cloud Composer, Cloud Run, and Pub/Sub to process new data into BigQuery

With help from SADA, my team is enabled to spend up to 30% less time on data wrangling and more time providing business intelligence and insights than before. This is what moves the needle and takes our business forward. They also helped us be smarter about how we will go about doing things in the future.

Asif Rahman | Vice President of Digital Analytics and Audience at AccuWeather