Axel, Super fast download manager for Linux

Tired by the speed of ordinary download managers?? Try axel. Lightweight , fast and efficient download manager. It is capable of juicing every bit of your available bandwidth and that means you don't have to wait too long to finish download. For more details click here

Batch download using wget

If I want to download all files in a directory of a website, i have to perform a long wget command, they is what I do at first:

wget -nd -r -l1 --no-parent http://www.foo.com/mp3/

This works fine, but sometimes it gives some silly index.@xx files that wish I do not want. To make your directory clean, if you knows the file format you want, you can do this:

wget -nd -r -l1 --no-parent -A.mp3 -A.wma http://www.foo.com/mp3/

I am going to explain briefly what is the meaning of the options I speficied.
-nd no directory, by default wget creates a dir
-r recursively download
-l1 (L one) level 1, download only of that particular folder, don’t go depth on it.
–no-parent I definately don’t want the parent’s files

Using wget through proxy servers

To use wget through proxies.
Set the environment variable using the following command
export http_proxy="http://proxy.example.com:8080"

Similarly or ftp
export ftp_proxy="http://proxy.example.com:8080"

For proxies that require authentication use --proxy-user=proxyusename --proxy-password=urpassword switch along with the wget command.
For example wget --proxy-user=name --proxy-password=password http://somelocation/index.html

For Details visit this site



wget: Download entire websites easy

wget is a nice tool for downloading resources from the internet. The basic usage is wget url:

wget http://linuxreviews.org/

Therefore, wget (manual page) + less (manual page) is all you need to surf the internet. The power of wget is that you may download sites recursive, meaning you also get all pages (and images and other data) linked on the front page:

wget -r http://linuxreviews.org/

But many sites do not want you to download their entire site. To prevent this, they check how browsers identify. Many sites refuses you to connect or sends a blank page if they detect you are not using a web-browser. You might get a message like:

Sorry, but the download manager you are using to view this site is not supported. We do not support use of such download managers as flashget, go!zilla, or getright

Wget has a very handy -U option for sites like this. Use -U My-browser to tell the site you are using some commonly accepted browser:

  wget  -r -p -U Mozilla http://www.stupidsite.com/restricedplace.html

The most important command line options are --limit-rate= and --wait=. You should add --wait=20 to pause 20 seconds between retrievals, this makes sure you are not manually added to a blacklist. --limit-rate defaults to bytes, add K to set KB/s. Example:

wget --wait=20 --limit-rate=20K -r -p -U Mozilla http://www.stupidsite.com/restricedplace.html

A web-site owner will probably get upset if you attempt to download his entire site using a simple wget http://foo.bar command. However, the web-site owner will not even notice you if you limit the download transfer rate and pause between fetching files.

Use --no-parent

--no-parent is a very handy option that guarantees wget will not download anything from the folders beneath the folder you want to acquire. Use this to make sure wget does not fetch more than it needs to if just just want to download the files in a folder.

Source: here

Password-less login with ssh

These instructions explain what to do in order to be able to use ssh for logging on a remote machine without entering your password. All our machines use ssh protocol 2, so please follow the corresponding instructions. However, for completeness, the instructions for protocol 1 are provided here as well. Please note that all instructions pertain to OpenSSH.

Protocol 2

On the originating machine, say

ssh-keygen -t rsa

Save the key in the default file (~/.ssh/id_rsa) and do not use a passphrase. This will create a file ~/.ssh/id_rsa.pub. Add the contents of this file to the file ~/.ssh/authorized_keys2 on the remote machine (i.e., the machine on which you want to log on). Whereas the id_rsa.pub file is world readable, the authorized_keys2 file should only be readable by you. Use chmod to set the permissions accordingly.

Protocol 1 (less secure, so consider this obsolete)

On the originating machine, say

ssh-keygen

Save the key in the default file (~/.ssh/identity) and do not use a passphrase. This will create a file ~/.ssh/identity.pub. Add the contents of this file to the file ~/.ssh/authorized_keys on the remote machine (i।e., the machine on which you want to log on). Whereas the identity.pub file is world readable, the authorized_key file should only be readable by you. Use chmod to set the permissions accordingly.

Source : here