[SOLVED] Python script in background downloading images, hangs up. How to keep downloading script running?

Issue

I'm running a python script (downloader.py) in the background on a data collection server. It goes through a list of links and downloads each of the images. (code snippet below) Since there are many images, I am running "nohup python downloader.py &" for no hangup and in the background.

Everything begins fine, but after some arbitrary amount of images, the images stop downloading. The process PID is still shown as a running process when I run "ps aux | grep downloader.py", but the nohup.out output file has just printed the link of the next file, as outputted by the "print(myfile + " " + link)" in the code snippet. (no error message whatsoever shown in the nohup.out, it just stops downloading or proceeding the next images in the link file). What could be the problem? The "nice" priority of the process in unix is set to 0.

for link in read_lines("data_links/"+myfile):
    try:
        print(myfile + " " + link)
        counter = counter + 1
        ##downloading the image and saving the file in data_collection/
        print("set file structure for download")
        f = open("data_collection/" + myfile.replace("_links.txt", "") + "/" + str(counter) + ".jpg", 'wb')
        print("beginning url request")
        f.write(request.urlopen(link).read())
        print("url request done, closing file")
        f.close()
        print("done downloading, moving onto next")
    except:
        print("downloading error, but no problem, we're moving on to the next one")
        continue

Solution

You're violating a fundamental tenet - make your try/except code as specific as possible. Right now, you're catching every single exception, and then continueing, regardless of what the error is, which is a very bad idea. Tailor your except clause to a specific error that could occur, then handle it appropriately.

Answered By - MattDMo