Stack Overflow archive
2 scoreaccepted

how do I download a large number of zip files with wget to a url

score
2
question views
2.7K
license
CC BY-SA 3.0

You need to get the redirect from bit.ly and then download all files. This is real ugly, but it worked:

bash
wget http://bitly.com/nuvi-plz --server-response -O /dev/null 2>&1 | \
  awk '(NR==1){SRC=$3;} /^  Location: /{DEST=$2} END{ print SRC, DEST}' | sed 's|.*http|http|' | \
while read url; do 
  wget -A zip -r -l 1 -nd $url -P test/files/downloads
done

If you use the direct link, this will work:

bash
wget -A zip -r -l 1 -nd http://feed.omgili.com/5Rh5AMTrc4Pv/mainstream/posts/ -P test/files/downloads

Originally posted on Stack Overflow. Public user contributions are licensed under Creative Commons Attribution-ShareAlike.