Friday, January 26, 2024

[SOLVED] bash for loop with error when detecting non 200 response from curl

Issue

I got a bash script which reads URLs from different .txt files (ven Array defines the txt files) and I curl them and if the Response Code is not 200 it writes an error in error.txt

Sadly it always does it at the end, even if there is no error in any of the links, any idea why?

for i in "${ven[@]}"; do
        while IFS='' read -r line || [[ -n "$line" ]]; do
                        IP=$(curl --write-out '%{http_code}' --silent --output /dev/null $line?upstream=$1)
                                if [ $IP != 200 ]
                                then
                                counter=$((counter + 1))
                                echo $(date +"%d.%m.%y %T") : $line >> error.txt
                                fi
        done < $i
done

Solution

Your script has some quoting issues, although the main issue is the inconsistent use of test expression brackets (double vs. single) and how you are checking for the number. 

if [[ "$IP" -ne 200 ]]

-ne means "not equal", and since you've already used double brackets stay consistent.

The other stuff is more preventative in the way you quote the variables:

for i in "${ven[@]}"; do
    while IFS='' read -r line || [[ -n "$line" ]]; do
        IP=$(curl --write-out '%{http_code}' --silent --output /dev/null "$line?upstream=$1")
        if [[ "$IP" -ne 200 ]]; then
            counter=$((counter + 1))
            echo "$(date +'%d.%m.%y %T')" : "$line" >> error.txt
        fi
    done < "$i"
done

NOTE: If a site is redirecting ( 301 ) then it will show an error — something to maybe consider.



Answered By - l'L'l
Answer Checked By - Willingham (WPSolving Volunteer)
at