[SOLVED] MWAA to start and stop EC2 Instances

Issue

I'm new to using the AWS managed Airflow service. I want to use Airflow to start an EC2 instance, make sure it's running and then do some further work in the instance.

So far i have this dag below which is basically a copy of this.

This however fails everytime and i'm not adept enough to know why?

import os
from datetime import datetime

from airflow import DAG
from airflow.models.baseoperator import chain
from airflow.providers.amazon.aws.operators.ec2 import EC2StartInstanceOperator, EC2StopInstanceOperator
from airflow.providers.amazon.aws.sensors.ec2 import EC2InstanceStateSensor

INSTANCE_ID = os.getenv("INSTANCE_ID", "instance-id")

with DAG(
    dag_id='example_ec2',
    schedule_interval=None,
    start_date=datetime(2021, 1, 1),
    tags=['example'],
    catchup=False,
) as dag:
    # [START howto_operator_ec2_start_instance]
    start_instance = EC2StartInstanceOperator(
        task_id="ec2_start_instance",
        instance_id=INSTANCE_ID,
    )
    # [END howto_operator_ec2_start_instance]

    # [START howto_sensor_ec2_instance_state]
    instance_state = EC2InstanceStateSensor(
        task_id="ec2_instance_state",
        instance_id=INSTANCE_ID,
        target_state="running",
    )
    # [END howto_sensor_ec2_instance_state]

chain(start_instance, instance_state)

Solution

Just going to answer my own question in case anyone else finds this.

The line below doesnt work for whatever reason

INSTANCE_ID = os.getenv("INSTANCE_ID", "instance-id")

The alternative is to do the following

from airflow.models import Variable # import Variable

INSTANCE_ID = Variable.get("INSTANCE_ID") # get INSTANCE_ID

Make sure you have added INSTANCE_ID as a variable by going to Admin -> Variable.

Answered By - Musa

Answer Checked By - Terry (WPSolving Volunteer)