[SOLVED] Using a shell script to replace tokens in a file

Issue

I have a file that holds token variables. When we switch environments we want to replace those tokens correctly for the environment we're in.

How can I do this in a Linux shell? I'm thinking awk or sed.

Example:

File has this data:

DB_HOST=__HOST__
DB_PASSWORD=__PASSWORD__

I want to read this file, recognize the __HOST__ and replace it with the environment variable $(HOST). We would get the environment variable name from the matched string between the two underscores.

Does anyone have a good idea how we could do this?

Solution

The program envsubst was designed for roughly this purpose, but it wants to to use standard shell format for the strings to be substituted from the environment:

# Here-doc delimiter quoted to avoid having the shell do the substitutions
$ HOST=myhost PASSWORD=secret envsubst <<"EOF"
DB_HOST=$HOST
DB_PASSWORD=${PASSWORD}
EOF

DB_HOST=myhost
DB_PASSWORD=secret

envsubst is part of Gnu gettext, and is widely available. If you really need to use the __...__ syntax, it would be easy to change it to envsubst syntax by piping through sed:

$ cat myfile.txt
DB_HOST=__HOST__
DB_PASSWORD=__PASSWORD__

$ sed -E 's/__(([^_]|_[^_])*)__/${\1}/g' myfile.txt |envsubst

Alternatively, if you have Python, there is a very simple solution:

from sys import stdin
from os import getenv
import re

pattern = re.compile(r"__(\w*)__")

def replacer(m):
    val = getenv(m[1])
    if val is None:
        # No such environment
        # Perhaps produce a warning, or an error
        # Here we just pass through the string unaltered
        return m[0]
    return val

print(pattern.sub(replacer, stdin.read()))

Example:

$ export HOST=myhost PASSWORD=secret 
$ python3 subst.py < myfile.txt
DB_HOST=myhost
DB_PASSWORD=secret

The key to this solution is that Python (and only a few other languages, unfortunately) allows the replacement argument of pattern substitution to be a function, which is then called on each successive match to produce the replacement text. That makes writing functions like this easier in Python than in awk, for example, where the replacement text is fixed.

Answered By - rici

Answer Checked By - Senaida (WPSolving Volunteer)