Monitor Apache Airflow with StatsD | Prometheus | Grafan | part1
#apacheairflow #airflowforbeginners #airflowmetrics #statsd #prometheus #grafana
In this video, you'll learn about how you can monitor airflow cluster by taking advantages of useful airflow statsd metrics. As part of single docker-compsoe file, we will spin up following services:
1. airflow-init: initializes airflow database schema and creates admin user
2. airflow-scheduler: runs airflow scheduler in LocalExecutor
3. airflow-webserver: runs airflow webserver exposing on port 8081
4. mysql: runs mysql database for airflow metadata
5. statsd-exporter: service that converts airflow statsd metrics to Prometheus metrics
6. Prometheus: runs Prometheus service that scraps airflow metrics from statsd-exporter
7. Grafana: runs Grafana instance where our final dashboards will be available.
---------------------------------------------------
👍 If this video was helpful to you, please don't forget to thumbs-up,
⌨️ leave a comment and
🔗 share with your friends.
⭐ Support our work by hitting that subscribe button.
🔔 hit bell icon to stay alert on more upcoming interesting stuff.
---------------------------------------------------
Timeline:
00:00 Intro
00:56 Monitoring data flow
01:41 Airflow metrics configs
02:01 Write docker-compose file
03:15 writing statsd-exporter service
04:02 statsd config mapping
06:01 test metrics
08:42 outro
-----------------------------------------------
Get complete code from here;
https://github.com/maxcotec/Apache-Ai...
Grafana and StatsD configs
https://github.com/databand-ai/airflo...
Statsd python client
https://github.com/jsocol/pystatsd
Watch the previous video here:
Run Airflow locally via Docker
• Build your first pipeline DAG | Apach...
Learn Something New Below:
https://maxcotec.com/blog/pv-center-f...
https://maxcotec.com/blog/5-cool-tech...
Смотрите видео Monitor Apache Airflow with StatsD | Prometheus | Grafan | part1 онлайн, длительностью часов минут секунд в хорошем качестве, которое загружено на канал MaxcoTec Learning 23 Сентябрь 2021. Делитесь ссылкой на видео в социальных сетях, чтобы ваши подписчики и друзья так же посмотрели это видео. Данный видеоклип посмотрели 17,595 раз и оно понравилось 189 посетителям.
