[SOLVED] Airflow: How to SSH and run BashOperator from a different server

Issue

Is there a way to ssh to different server and run BashOperator using Airbnb's Airflow? I am trying to run a hive sql command with Airflow but I need to SSH to a different box in order to run the hive shell. My tasks should look like this:

1. SSH to server1
2. start Hive shell
3. run Hive command

Thanks!

Solution

NOT available for airflow 2.x.

I think that I just figured it out:

1. Create a SSH connection in UI under Admin > Connection. Note: the connection will be deleted if you reset the database

2. In the Python file add the following

    from airflow.contrib.hooks import SSHHook
    sshHook = SSHHook(conn_id=<YOUR CONNECTION ID FROM THE UI>)
   

3. Add the SSH operator task

    t1 = SSHExecuteOperator(
        task_id="task1",
        bash_command=<YOUR COMMAND>,
        ssh_hook=sshHook,
        dag=dag)
   

Thanks!

Answered By - CMPE

Answer Checked By - David Marino (WPSolving Volunteer)