Apache Airflow | Build Custom Sensor | KubernetesPodSensor

Published: 01 January 1970
on channel: MaxcoTec Learning

717

In this video we will build a custom sensor for Apache Airflow known as KubernetesPodSensor. This Sensor is inspired from KubernetesPodOperator, which lets you run a docker image in a dedicated pod on each poke interval.

Prerequsites:
1. Airflow Sensors:    • Airflow sensors | Create custom senso...
2. Airflow on Kubernetes:    • Airflow with Kubernetes Executors | R...
2.1 Airflow on Kubernetes using rancher:    • Run Airflow on kubernetes using Ranch...
3. Airflow KubernetesPodOperators:    • Deep dive into Airflow Kubernetes Pod...

Get the code:
dynamoDB-readiness-check: https://github.com/maxcotec/dynamodb-...

dag and sensor code: https://github.com/maxcotec/airflow-k...

Timeline:
00:00 intro & outline
00:38 prerequisites
01:31 Airflow Sensor recap
02:08 why KubernetesPodSensor
02:43 How KubernetesPodSensor will work
04:13 write code for sensor
06:28 write dag code
09:23 build image
11:06 deploy airflow
12:07 test the dag
15:02 conclusion
15:53 outro

#apacheairflow #maxcoteclearning #apacheairflowsensor

Watch video Apache Airflow | Build Custom Sensor | KubernetesPodSensor online, duration hours minute second in high quality that is uploaded to the channel MaxcoTec Learning 01 January 1970. Share the link to the video on social media so that your subscribers and friends will also watch this video. This video clip has been viewed 717 times and liked it 8 visitors.

11,720

318