Note
Click here to download the full example code
12.3.10.5.2. Thread pool executorΒΆ
Asyncio
/concurrent
heavily changed from python 3.4
to 3.7
, better read the docs
and do some tutorials. Asyncio is preferred over plain concurrent module.
import concurrent.futures
import urllib.request
URLS = [
"http://www.foxnews.com/",
"http://www.cnn.com/",
"http://europe.wsj.com/",
"http://www.bbc.co.uk/",
"http://some-made-up-domain.com/",
]
Retrieve a single page and report the url and contents
def load_url(url, timeout):
with urllib.request.urlopen(url, timeout=timeout) as conn:
return conn.read()
We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
# Start the load operations and mark each future with its URL
future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
try:
data = future.result()
except Exception as exc:
print("%r generated an exception: %s" % (url, exc))
else:
print("%r page is %d bytes" % (url, len(data)))
'http://www.foxnews.com/' page is 311539 bytes
'http://www.bbc.co.uk/' page is 449682 bytes
'http://www.cnn.com/' page is 1143482 bytes
'http://some-made-up-domain.com/' page is 1127 bytes
'http://europe.wsj.com/' generated an exception: HTTP Error 403: Forbidden
Total running time of the script: ( 0 minutes 0.870 seconds)