At SADA, we implement Google Cloud Composer for our clients who require powerful and intricate workflow management capabilities for their data pipeline. Whether that data pipeline is for Bigquery, Dataproc, Dataflow, or extract, transform, and load (ETL) workflows, Google Cloud Composer offers a managed Apache Airflow-based workflow management solution. Cloud Composer natively integrates with Google Cloud Platform (sometimes referred to as GCP) and is a managed service, so you don’t need to install or manage Apache Airflow.
One challenge we have encountered in Google Cloud Composer is monitoring workflow failures, specifically around directed acyclic graphs (DAGs) and tasks. Cloud Monitoring identifies and alerts on various events in Google Cloud Composer, but DAG and tasks are omitted and require explicit monitoring configurations.
In this article, we will share those configurations so you can effectively monitor your Google Cloud Composer DAGs and tasks, both via the Google Cloud Console or by our preferred method via Infrastructure as Code (IaC) using Terraform.