Issue
I'm a linux user for more than 10 year and have been learning to use airflow from some tutorials.
But I have made such a big mess on my OS, to the point I could not even stop airflow from startup on boot. I could not run any dag that I have made, could not uninstall it. Could not even use it in a virtual enviromment, because there was another airflow on port 8080 (as I said, I did a lot of tutorials). So on...
So I decided to make a clean linux reinstall and start from scratch. And I want some roadmap to not make those mistakes again.
I have some experience in virtual eviromment from using with python. I know the basics of Docker.
I'm confused about should the airflow run inside a docker? Or the docker runs inside the airflow?
If I run airflow outside docker, should the airflow (with all the pip packages) be installed inside a virtual enviromment?
What should I learn before airflow?
What would be the roadmap to run a simple Bash and Python Operaror?
Solution
I am using the docker compose solution. Airflow as docker container (worker and scheduler).
Follow the instructuons from airflow official page.
Another option is to use Astronomer solution for airflow that wrappes airflow deploy with thier cli tool and many video guides.
Answered By - ozs Answer Checked By - Willingham (WPSolving Volunteer)