ASK ASTRO

Many data teams leverage Databricks' optimized Spark engine to run heavy workloads like machine learning models, data transformations, and data analysis. While Databricks offers some orchestration with Databricks Workflows, they are limited in functionality and do not integrate with the rest of your data stack. Using a tool-agnostic orchestrator like Airflow gives you several advantages, like the ability to: - Use CI/CD to manage your workflow deployment. Airflow DAGs are Python code, and can be [integrated with a variety of CI/CD tools](https://docs.astronomer.io/astro/ci-cd- templates/template-overview) and tested. - Repair single tasks in your Databricks Workflow. If a task fails, you can re-run it without having to re-run the entire Workflow, saving valuable compute resources. - Use [task groups](task- groups.md) within Databricks Workflows, enabling you to collapse and expand parts of larger Databricks Workflows visually. - Leverage Airflow cross-DAG dependencies to trigger Databricks Workflows from other DAGs in your Airflow environment, allowing for a data-driven architecture. - Use familiar Airflow code as your interface to orchestrate Databricks notebooks as Workflows. - Inject parameters into your Databricks Workflow at the Workflow-level. These parameters can be dynamic and retrieved at runtime from other Airflow tasks.

Airflow™ is a batch workflow orchestration platform. The Airflow framework contains operators to connect with many technologies and is easily extensible to connect with a new technology. If your workflows have a clear start and end, and run at regular intervals, they can be programmed as an Airflow DAG. If you prefer coding over clicking, Airflow is the tool for you. Workflows are defined as Python code which means: - Workflows can be stored in version control so that you can roll back to previous versions - Workflows can be developed by multiple people simultaneously - Tests can be written to validate functionality - Components are extensible and you can build on a wide collection of existing components Rich scheduling and execution semantics enable you to easily define complex pipelines, running at regular intervals. Backfilling allows you to (re-)run pipelines on historical data after making changes to your logic. And the ability to rerun partial pipelines after resolving an error helps maximize efficiency. Airflow's user interface provides: > 1. In-depth views of two things: > > > i. Pipelines > > ii. Tasks > > 2. Overview of your pipelines over time From the interface, you can inspect logs and manage tasks, for example retrying a task in case of failure. The open-source nature of Airflow ensures you work on components developed, tested, and used by many other companies around the world. In the active community you can find plenty of helpful resources in the form of blog posts, articles, conferences, books, and more. You can connect with other peers via several channels such as Slack and mailing lists. Airflow as a Platform is highly customizable. By utilizing public- airflow-interface{.interpreted-text role="doc"} you can extend and customize almost every aspect of Airflow.

Airflow™ was built for finite batch workflows. While the CLI and REST API do allow triggering workflows, Airflow was not built for infinitely running event-based workflows. Airflow is not a streaming solution. However, a streaming system such as Apache Kafka is often seen working together with Apache Airflow. Kafka can be used for ingestion and processing in real-time, event data is written to a storage location, and Airflow periodically starts a workflow processing a batch of data. If you prefer clicking over coding, Airflow is probably not the right solution. The web interface aims to make managing workflows as easy as possible and the Airflow framework is continuously improved to make the developer experience as smooth as possible. However, the philosophy of Airflow is to define workflows as code so coding will always be required. ::: {.toctree hidden="" caption="Content"} Overview start installation/index security/index tutorial/index howto/index ui core- concepts/index authoring-and-scheduling/index administration-and- deployment/index integration public-airflow-interface best-practices faq Release Policies release_notes privacy_notice project license ::: ::: {.toctree hidden="" caption="References"} Operators and hooks CLI Templates Stable REST API deprecated-rest-api-ref Configurations Extra packages ::: ::: {.toctree hidden="" caption="Internal DB details"} Database Migrations Database ERD Schema :::

Running Jupyter notebooks from Airflow is a great way to accomplish many common data science and data analytics use cases like generating data visualizations, performing exploratory data analysis, and training small machine learning models. However, there are several cases where this might not be the best approach: - Because the Jupyter notebook runs within your Airflow environment, this method is not recommended for notebooks that process large data sets. For notebooks that are computationally intensive, Databricks or notebook instances from cloud providers like AWS or GCP may be more appropriate. - Notebooks are run in their entirety during each DAG run and do not maintain state between runs. This means you will run every cell in your notebook on every DAG run. For this reason, if you have code that takes a long time to run (such as a large ML model), a better approach may be to break up the code into distinct Airflow tasks using other tools.

To get the most out of this tutorial, make sure you have an understanding of: - The basics of Databricks. See Getting started with Databricks. - Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](get- started-with-airflow.md). - Airflow operators. See [Operators 101](what-is- an-operator.md). - Airflow connections. See Managing your Connections in Apache Airflow.

--- title: "ELT with Airflow and Databricks" description: "Use Airflow, Databricks and the Astro Python SDK in an ELT pipeline to analyze energy data." id: use-case-airflow-databricks sidebar_label: "ELT with Airflow + Databricks" sidebar_custom_props: { icon: 'img/integrations/databricks.png' } --- Databricks is a popular unified data and analytics platform built around fully managed Apache Spark clusters. Using the [Astro Databricks provider package](https://github.com/astronomer/astro- provider-databricks), you can create a Databricks Workflow from Databricks notebooks and run the Databricks Workflow in an Airflow DAG. This lets you use Airflow's orchestration features in combination with Databricks' cheapest compute. To get data in and out of Databricks, you can use the open-source [Astro Python SDK](https://astro-sdk- python.readthedocs.io/en/stable/index.html), which greatly simplifies common ELT tasks like loading data and creating pandas DataFrames from data in your warehouse. This example uses a DAG to extract data from three local CSV files containing the share of solar, hydro and wind electricity in different countries over several years, run a transformation on each file, load the results to S3, and create a line chart of the aggregated data. After the DAG runs, a graph appears in the include directory which shows the combined percentage of solar, hydro and wind energy in a country you selected. :::info For more detailed instructions on using Databricks with the Astro Databricks provider, see the Databricks tutorial. :::