wget
is a non-interactive downloaderwget [OPTIONS] URL
wget www.umbc.edu
head index.html.1
wget
¶wget
, and many other command line tools can theoretically launch 100s of request a secondwget
has many options to prevent this if you are downloading multiple files at oncewget --mirror ‐‐page-requisites \
‐‐convert-links ‐‐adjust-extension \
-P./local_443-2 --wait 1 --random-wait \
https://www.csee.umbc.edu/~bwilk1/433/
wget
Features¶wget
allows you specify a list of urls to download by using the -i flagwget
to download the websitessed
and other tools to strip the text out from the website# Get lsit of addresses from "https://univ.cc/search.php?dom=edu&key=&start=1"
# Extract the URLS
mapfile sites_to_get < targets
# Process files using wget and sed
# Get URL
# Get School Name
cat Abilene_Christian_University.txt
curl
¶curl
is a more powerful tool that allows uploading and download overcurl
prints to STDOUTcurl http://www.umbc.edu
curl -I http://www.umbc.edu
curl
allows POST by using the -X
flagcurl -X POST -d "DATA" URL
bash
¶#!/bin/bash FLAGS
cat src/shell/syntax_example.sh
./src/shell/syntax_example.sh
cat src/shell/syntax_error_example.sh
./src/shell/syntax_error_example.sh
cat ./src/shell/cla_debug.sh
./src/shell/cla_debug.sh Arg1 SOmething goes here