Issue
I am writing a python program, and when I test it, at the first 20 minutes, it works fine, then suddenly the memory consumption increase a lot from 1G to more than 6G, then it ended with no error. I guess it might be killed by kernel because both memory and SWAP are full. Here is my question, 1. how do I know why it quit? 2. Is there anyway I could know why it suddenly consumes so much memory? The code is more than 800 lines and I don't which part I could simplified, so I didn't paste it here. Basically the program is just a big loop doing the same thing to different data.
Here is the last moment before it crashes, 
I don't know the why the VIRT suddenly goes so high, but before this happens, there is also some strange phenomenon, since the whole program is using 30 threads with proxies to crawl some websites, the normal traffic is around 1MB/s, but traffic before this happens is around 10MB/s, even 20MB/s (the speed is possible, since I am testing on a VPS), but traffic when this is happening is almost 0. I don't know how to link the traffic and memory problem together.
Solution
You can start with Python line profiler, it is used to see how fast and how often each line of code is running in your script.
pip install line_profiler
Once you time it, you can start analyzing memory
pip install -U memory_profiler
pip install psutil
The quickest way to find “memory leaks” is to use an awesome tool called objgraph This tool allows you to see the number of objects in memory and also locate all the different places in your code that hold references to these objects.
pip install objgraph
Answered By - gogasca
