Airflow sensors | Create custom sensor to sense files

Published: 22 July 2023
on channel: MaxcoTec Learning

1,723

See how airflow sensors can pitch in your ETL pipelines to sense something before proceeding with downstream dependencies.

Get code here:
https://github.com/maxcotec/Apache-Ai...

How to deploy airflow locally using Docker
   • Run Airflow 2.0 via Docker | Minimal ...

How to deploy airflow locally on a multinode kubernets cluster
   • Airflow with Kubernetes Executors | R...

build your first data pipeline in airflow
   • Build your first pipeline DAG | Apach...

#apacheairflow #kubernetes #airflowsensors #dataengineering #etl #maxcoteclearning

Video outline:
00:00 intro and outline
00:38 definition
01:27 operation modes
02:23 sensor types
02:57 github repo & reference videos
03:47 spin-up airflow locally
05:15 write dag with BashSensor
06:42 test BashSensor
07:18 write custom sensor - check file exists
09:19 test custom file-check sensor
10:30 outro

Watch video Airflow sensors | Create custom sensor to sense files online, duration hours minute second in high quality that is uploaded to the channel MaxcoTec Learning 22 July 2023. Share the link to the video on social media so that your subscribers and friends will also watch this video. This video clip has been viewed 1,723 times and liked it 21 visitors.

11,720

318