The joy of the linux/unix command line is how versatile the commands are. I recently had 50,000 URL’s I needed to download in a text file. I was thinking about writing a crawler in Python to do it but ended up just doing the following,
cat urllist | xargs -P16 wget -i
A 16 thread (process really) webcrawler in a single command. Joy.