Thursday, July 28, 2022

[SOLVED] AWS SQS not receiving SNS messages

Issue

I created a SNS topic that publishes all the information coming out of Cloudformation via the cli. However, when I check the queue, it is not receiving any of the SNS messages. I verified the SNS is working by subscribing my email to it, so the issue seems to be in the connection between the queue and the SNS. However, I cannot find any problems with my syntax. I, as far as I know, have followed amazon's documentation precisely.

Bash:

#SNS parameters
SNS_NAME="${NAME}_SNS"
SQS_NAME="${NAME}_SQS"

#Create SNS topic to send cloudformation notifications to
SNS_ARN=`aws sns create-topic --name ${SNS_NAME} | jq -r '.TopicArn'`

#Create SQS to send SNS to (holding SNS messages for lambda -^ up)
SQS_URL=`aws sqs create-queue --queue-name ${SQS_NAME} | jq -r '.QueueUrl'`
SQS_ARN=`aws sqs get-queue-attributes --queue-url ${SQS_URL} --attribute-names QueueArn | jq -r '.Attributes .QueueArn'`

#subscribe the queue to the notifications
aws sns subscribe --topic-arn ${SNS_ARN} --protocol sqs --notification-endpoint ${SQS_ARN}
aws sns subscribe --topic-arn ${SNS_ARN} --protocol email-json --notification-endpoint ${EMAIL}

#Create the stack which kicks everything else off-
aws cloudformation create-stack $REGIONTEXT $ITYPETEXT --capabilities CAPABILITY_IAM --template-url https://${BUCKETNAME}.s3.amazonaws.com/${TEMPLATE} --notification-arns ${SNS_ARN} --stack-name $NAME --parameters ParameterKey=SNSARN,ParameterValue=${SNS_ARN} ParameterKey=Bucket,ParameterValue=${BUCKETNAME} ${PARAMTEXT} ${EXTRAARGS}

Solution

Thank you to Mark B for his answer. It provided the start to getting this working. However, in order to make a policy document work via the CLI there are a few quirks that aren't covered in the docs.

  1. There are all sorts of errors trying to pass json directly to the --attributes flag in aws sqs set-queue-attributes command. For some reason it requires the modifying json to be in a .json document referenced by the cli.
  2. In the .json file provided to the cli, all of the double quotes inside the "Policy" value (nested json) must be escaped (i.e. { \"Statement\": \"HelloWorld\" }). If this is not followed, it will validation errors. I ended up needing to use the ascii escape characters in order properly format the output (\x5C).
  3. The json file must be referenced by using file://local-location in the --attributes flag. It throws errors if this is not followed.

See the following elements I used for reference:

load_sqs.sh:

SQS_POLICY=
sqs-policy()
{
#First param is the queue arn, second param is the topic arn
SQS_POLICY=`printf '{ "Policy": "{\x5C\"Version\x5C\":\x5C\"2012-10-17\x5C\",\x5C\"Statement\x5C\":[{\x5C\"Sid\x5C\":\x5C\"CloudformationLambdaSQSPolicy\x5C\",\x5C\"Effect\x5C\":\x5C\"Allow\x5C\",\x5C\"Principal\x5C\":\x5C\"*\x5C\",\x5C\"Action\x5C\":\x5C\"sqs:SendMessage\x5C\",\x5C\"Resource\x5C\":\x5C\"%s\x5C\",\x5C\"Condition\x5C\":{\x5C\"ArnEquals\x5C\":{\x5C\"aws:SourceArn\x5C\":\x5C\"%s\x5C\"}}}]}" }' "$1" "$2"`
`echo $SQS_POLICY > $PWD/sqs-policy.json`
}

#SNS parameters
SNS_NAME="${NAME}_SNS"
SQS_NAME="${NAME}_SQS"

#Create SNS topic to send cloudformation notifications to
SNS_ARN=`aws sns create-topic --name ${SNS_NAME} | jq -r '.TopicArn'`

#Create SQS to send SNS to (holding SNS messages for lambda -^ up)
SQS_URL=`aws sqs create-queue --queue-name ${SQS_NAME} | jq -r '.QueueUrl'`
SQS_ARN=`aws sqs get-queue-attributes --queue-url ${SQS_URL} --attribute-names QueueArn | jq -r '.Attributes .QueueArn'`

#Add necessary SQS <--> SNS permissions
sqs-policy ${SQS_ARN} ${SNS_ARN}
`aws sqs set-queue-attributes --queue-url ${SQS_URL} --attributes file://sqs-policy.json`

#subscribe the queue to the notifications
aws sns subscribe --topic-arn ${SNS_ARN} --protocol sqs --notification-endpoint ${SQS_ARN}

sqs-policy.json:

{ "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"CloudformationLambdaSQSPolicy\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"sqs:SendMessage\",\"Resource\":\"ResourceARN\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"SourceARN\"}}}]}" }


Answered By - asdf
Answer Checked By - Timothy Miller (WPSolving Admin)
at