Search

bash download from url list into directories

bash download from url list into directories

Prerequisite:
the file specified by urlFile must exist and contain a list of URLs, one per line.
baseDir is where the files will end up. It will be created if it does not exist.
splitText is the text that will be split in each URL
example:

http://www.example.com/textfiles/subdir1/subdir2/filename.txt

#!/bin/bash
#user serviceable area, change to your needs
urlFile="../urllist.txt"
baseDir="files/"
#end user serviceable area, change to your needs

#example URL for below: http://www.example.com/textfiles/subdir1/subdir2/filename.txt

cat $urlFile | while read line
do
if [ "$line" != "" ]; then
a="${line#*//}" # a=textfiles/subdir1/subdir2/filename.txt
fileName="${line#*//*/}" # fileName=filename.txt
path=$"{fileName%/*}" # path=textfiles/subdir1/subdir2

if [ "$path" != "" ]; then
mkdir -p "$baseDir$path"
path="$path/"
fi
echo Downloading $line
wget -q "$line" -O "$baseDir$path$d"
fi
done

Related posts

Leave a Comment