Issue

I have an algorithm which mask data of a column entered by user. The below function has been designed for the same. when script was executed , this took around 4 hrs to mask 100 thousands record. client want the same in 10 min. Could you please suggest how can I implement the below so that its performance can be improved. Is there anyway to change function without changing the algorithm.

data_mask() {
col_val=$1
l_ret_str=""
l_an=0
l_lp=0
l_mod=0
absnum=0
austart=65
auend=90
aclsize=26
alstart=97
alend=122
nstart=48
nend=57
nclsize=10
l_lp=`expr length "$col_val"`
if [[ $l_lp -ne 0 ]]; then
for i in `eval "echo {1..$l_lp}"`
do
single_char=$(SUBSTR "$col_val" $i)
ascii_num_val=$(ASCII "$single_char")
l_mod=$((l_mod+ascii_num_val))
done
l_mod=$((l_mod % nclsize))
for i in `eval "echo {1..$l_lp}"`
do
single_char=$(SUBSTR "$col_val" $i)
ascii_num_val=$(ASCII "$single_char")
l_an=$ascii_num_val
tempvar=$((l_an - l_lp - l_mod - i))
absnum=$(ABS $tempvar)
if [[ $l_an -ge $austart && $l_an -le $auend ]]; then
tempmodval=$((absnum % aclsize))
tempasciival=$((austart + tempmodval))
l_ret_str=$l_ret_str$(CHR $tempasciival)
elif [[ $l_an -ge $alstart && $l_an -le $alend ]]; then
tempmodval=$((absnum % aclsize))
tempasciival=$((alstart + tempmodval))
l_ret_str=$l_ret_str$(CHR $tempasciival)
elif [[ $l_an -ge $nstart && $l_an -le $nend ]]; then
tempmodval=$((absnum % nclsize))
tempasciival=$((nstart + tempmodval))
l_ret_str=$l_ret_str$(CHR $tempasciival)
else
tempmodval=$((absnum % nclsize))
tempasciival=$((austart + tempmodval))
l_ret_str=$l_ret_str$(CHR $tempasciival)
fi
done
fi
echo "$l_ret_str"
}

Here col_val=$1 entered by user.if user enters 2 then our code will mask second column. we are calling above function through below.

    while read p; do
if [[ $line -le $skip_line ]]; then
echo "$p" >> $outputfile
else
pre_str=`echo $p | cut -d'|' -f1-$((colnum - 1))`
column_value=`echo $p | cut -d'|' -f$colnum`
post_str=`echo $p | cut -d'|' -f$((colnum + 1))-$totalcol`
echo "column_value=$column_value"
maskvalue=$(data_mask "$column_value")
echo $pre_str"|"$maskvalue"|"$post_str >> $outputfile
fi
line=$((line + 1))
done <$temp_outputfile

Here we are splitting the files in 3 parts. and then calling to our function. here skipline is the number of line our code should skip.eg header. so if input is

id|name|dept
11|Shrut|consultant
12|wipro|HR
13|capgemini|IT

then output should be like below.

id|name|dept
11|sqmbr|consultant
12|itzaw|HR
13|khvlipkoi|IT

Please suggest some way. if you need some clarification I will provide in comment but please dont put it on hold. I have to enhance the speed of execution without changing algorithm written in data_mask(). function can be changed but not the algorithm. I am expecting your help on the same.

Solution

Bash is not very performant doing loops. Since you tagged the question python, python should be ok?

def data_mask(col_val):
    mod = len(col_val) + sum(map(ord, col_val)) % 10
    result = ""
    for i, ch in enumerate(col_val, mod + 1):
        absnum = abs(ord(ch) - i)
        if 'A' <= ch <= 'Z':
            ch = chr(ord('A') + absnum % 26)
        elif 'a' <= ch <= 'z':
            ch = chr(ord('a') + absnum % 26)
        elif '0' <= ch <= '9':
            ch = chr(ord('0') + absnum % 10)
        else:
            ch = chr(ord('A') + absnum % 10)
        result += ch
    return result

while open(temp_outputfile) as lines:
    while open(outputfile, 'w') as output:
        output.write(next(lines))
        for line in lines:
            pre, col_val, post = line.split('|', 2)
            output.write("{}|{}|{}".format(pre, data_mask(col_val), post))

Answered By - Daniel

Monday, October 25, 2021

[SOLVED] Enhancement of unix script

Issue

Solution