[SOLVED] how to pass double braces from shell script to python script in spark?

Issue

I have a python script a.py which is run on spark. It is submitted by shell script b.sh. I need to pass a parameter which contains double braces from b.sh to a.py. The following is what I did.

In b.sh I wrote:

spark_submit=/usr/custom/spark/bin/spark-submit
$spark_submit \
--name "test" \
a.py --param "{'a':{'b':2}}"

In a.py I wrote:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--param", type=str) 
args = parser.parse_args()
print(args)

I wish to get Namespace(param="{'a':{'b':2}}"), however it returns Namespace(param="{'a':{'b':2"), and the double braces vanished. How can I solve this?

Solution

This appears to be caused by yarn, a tool used by spark with certain configurations. See the answers here: Scala - curly brackets string missing

Answered By - Hugo Rivera