[SOLVED] How to escape special characters in Terraform local-exec command line

Issue

I have the following terraform :

resource "null_resource" "cognito_user_pool_client_id" {

 triggers  =  { always_run = "${timestamp()}" }
 provisioner "local-exec" {
  command = "aws cognito-idp list-user-pool-clients --user-pool-id  ${aws_cognito_user_pool.main.id} --region ${var.region} | grep -o '"ClientId": "[^"]*' | grep -o '[^"]*$' > ${path.module}/cognito_app_client.txt"

  depends_on = ["aws_cognito_user_pool_domain.main"]
}

where I am trying to get the app client id from userpool in cognito. Anyway this line fails with terraform :

Error: Invalid character

  on kibana_cognito.tf line 163, in resource "null_resource" "cognito_user_pool_client_id":
 163:   command = "aws cognito-idp list-user-pool-clients --user-pool-id  ${aws_cognito_user_pool.main.id} --region ${var.region} | grep -o '"ClientId": "[^"]*' | grep -o '[^"]*$' > ${path.module}/cognito_app_client.txt"

This character is not used within the language.


Error: Invalid character

  on kibana_cognito.tf line 163, in resource "null_resource" "cognito_user_pool_client_id":
 163:   command = "aws cognito-idp list-user-pool-clients --user-pool-id  ${aws_cognito_user_pool.main.id} --region ${var.region} | grep -o '"ClientId": "[^"]*' | grep -o '[^"]*$' > ${path.module}/cognito_app_client.txt"

Single quotes are not valid. Use double quotes (") to enclose strings.


Error: Invalid character

  on kibana_cognito.tf line 163, in resource "null_resource" "cognito_user_pool_client_id":
 163:   command = "aws cognito-idp list-user-pool-clients --user-pool-id  ${aws_cognito_user_pool.main.id} --region ${var.region} | grep -o '"ClientId": "[^"]*' | grep -o '[^"]*$' > ${path.module}/cognito_app_client.txt"

This character is not used within the language.

This works in aws cli :

(venv) ➜  virginia git:(master) ✗ aws cognito-idp list-user-pool-clients --user-pool-id  123456 --region us-east-1 | grep -o '"ClientId": "[^"]*' | grep -o '[^"]*$'
dsewe24dwr2e

how to make this work in terraform ?

Solution

You have a few different "languages" here to deal with, each of which will possibly need some escaping to contain the one inside it, but since you asked about Terraform in particular I'll focus on that first layer of escaping, to make sure the shell sees the string you want it to see.

The relevant documentation for this part of Terraform language syntax is Strings and Templates, which talks about the two different kinds of strings in the Terraform language and what kinds of escaping are available in each of them.

---

You're currently using a quoted string, so you need to contend both with the quoted string escaping and the template escaping. Your desired string only interacts with the quoted string escaping, because it contains some literal quote characters " which you want to be handled by the shell rather than by Terraform, and so you can escape them as \" like the documentation suggests:

    command = "aws cognito-idp list-user-pool-clients --user-pool-id  ${aws_cognito_user_pool.main.id} --region ${var.region} | grep -o '\"ClientId\": \"[^\"]*' | grep -o '[^\"]*$' > ${path.module}/cognito_app_client.txt"

For completeness here I'll also say that Unix shells typically also support a ${NAME} syntax as an alternative to $NAME in situations where the latter would be ambiguous. ${NAME} conflicts with Terraform's template syntax and so if you want to include a sequence like that in the shell command like then you'd need to escape it as shown in the second table in the documentation, by writing $${NAME} instead.

---

If you read the following section on heredoc strings you'll see that one benefit they have is that they aren't delimited by quotes and so the contents are not interpreted for backslash escape sequences, and so you can write quotes and other special characters literally:

    command = <<-EOT
      aws cognito-idp list-user-pool-clients --user-pool-id  ${aws_cognito_user_pool.main.id} --region ${var.region} | grep -o '"ClientId": "[^"]*' | grep -o '[^"]*$' > ${path.module}/cognito_app_client.txt
    EOT

Terraform does still interpret template sequences ${ ... } and %{ ... } in here, so if you had literal ${NAME} sequences for the shell you'd still need to write them as $${NAME} in this context, but you don't need to escape the quotes anymore because this type of string is delimited by a longer, user-specified marker EOT, allowing quotes inside to be treated as literals.

One further advantage of using a heredoc string is that you can split the command over multiple lines, which might make it easier to read:

    command = <<-EOT
      aws cognito-idp list-user-pool-clients --user-pool-id  ${aws_cognito_user_pool.main.id} --region ${var.region} \
        | grep -o '"ClientId": "[^"]*' | grep -o '[^"]*$' > ${path.module}/cognito_app_client.txt
    EOT

Note that Unix shells typically require escaping a newline with \ to avoid interpreting it as two separate commands, so I included that in the above example, but from Terraform's perspective this is just a two-line string to be passed on to the shell, after template expansion is complete.

Answered By - Martin Atkins

Answer Checked By - Terry (WPSolving Volunteer)