This Question have no Text answers yet!

Python Requests in 1000 Threads

By : passwd
Source: Stackoverflow.com
Question!

I am trying to run Requests in approximately 1000 threads with this script:

openDictionaryFile = open(dictionaryFile, "r")
linesOfDictionary = openDictionaryFile.readlines()
openDictionaryFile.close()

for wordToScan in wordsList:
#...some code

    dictionaryFileQueue = Queue.Queue()

    for word in linesOfDictionary:
        dictionaryFileQueue.put(word.strip())

    for i in range(20):
        i_thread = Thread(target=words_Scan_Start, args=(dictionaryFileQueue, wordToScan))
        threads.append(i_thread)

for thread in threads:
    for wordThread in thread:
        wordThread.start()

def words_Scan_Start(dictionaryFileQueue, wordToScan):
    # Some code
    checkWord(url, wordToScan)
    # Some code

#maximum Open File Limit
maxOpenFileLimit = resource.getrlimit(resource.RLIMIT_NOFILE)[0]
# Making a ThreadLimiter based on system OpenFile limit
threadLimiter = BoundedSemaphore(maxOpenFileLimit-124)

# Skip SSL connection warnings
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Requests pool has MaxSize = 10. Here we increasing size of the pool to prevent [Errno -3] when some sockets are in CLOSE_WAIT state.
requestSessions = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_maxsize=(maxOpenFileLimit+1000))
requestSessions.mount('http://', adapter)
requestSessions.mount('https://', adapter)

# Sending Requests
def checkWord(url, wordToScan):
    global requestSessions
    headers = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0"}
    try:
        threadLimiter.acquire()
        resp = requestSessions.get(url+"/action.php?id="+wordToScan, timeout=60, verify=False, stream=False, headers=headers)
        #resp.close()
        threadLimiter.release()
        if (resp.status_code < 400):
            return "Exists" 
        else:
            return "Nothing"
    except requests.exceptions.Timeout:
        return "!Connection timed out"
    except Exception as e:
        return "!"+str(e)

Here is my question. When I'm starting this script and checking sockets with lsof -i|grep python at the begining of the scanning I have roughly 800-900 sockets. But after I'm getting over 900 (as I've made a limit in threadLimiter), then I'm getting roughly 80-150 after checking lsof -i|grep python. Can anyone tell if I'm doing it right? And why does the number of runned processes drastically reduced after threadLimiter.aquire()? Where are the other 800 processes? Am I doing it right connecting via Requests in 900 threads? Is there a better way?

By : passwd


Video about Python Requests in 1000 Threads