Fast Copy

The normal way
To copy files recursively from src1/, src2/, to dest/ you do

cp -Rv src1/ src2/ dest/

The piped way
cp uses character by character copy. Using the kernel pipes support, we could copy files block by block. tar converts the directories recursively into a single stream.At one end we create a single stream out of source directories and at other end we extract this stream and put it in the destination directory. The transfer between the two ends is by means of pipes. tar -Sc src1/ src2/ | tar -C dest/ -xv

This is not necessarily faster, but is much more flexible. Note that ext2, ext3 and most modern Linux file systems have a capability called 'sparse files'. This is used to store large files with lots of zeroed content in an efficient manner. Most of the time you should not care about this, but certain programs make extensive use of this feature (e.g. net-p2p/mldonkey). You should be careful when copying sparse files using this method, as your disk usage can explode if you forget to use the -S switch on the left side. Whereas the cp utility does handle sparse files automatically, tar (without -S) does not. Sparse files in the source directory will be stored in full representation on the destination directory.

Network with SCP
If you want to transfer a file, the following may be used scp -C remotebox:path/to/sourcefile. scp -C localfile remotebox:path/to/destination/
 * 1) Remote to local
 * 1) Local to remote

If multiple files or directories are to be transferred, the following may be used scp -rC remotebox:path/to/sourcedir. scp -Cr localdir remotebox:path/to/destination/
 * 1) Remote to local
 * 1) Local to remote

Network with SSH
A distinction must be whether from a local system to a remote or vice versa is to be copied:

tar czv dir/files | ssh remote.box.com "tar xz -C /dir/" tar czv ListOfFiles | ssh -c blowfish remote.box.com tar xz -C /home/user/PathToCopy ssh remote.box.com tar cz -C BeginDirCopyFiles |tar xz -C DirToCopy
 * 1) to the remote system
 * 1) to the remote system with faster encryption
 * 1) from remote to local

Transfer of individual files can also be made with tar, but also suitable for this cat: cat dir/file | ssh -C remote.box.com "cat > /dir/file" cat dir/file | gzip | ssh -C remote.box.com "gunzip > /dir/file" cat dir/file | bzip2 | ssh -C remote.box.com "bunzip2 > /dir/file" ssh remote.box.com "cat /dir/file" > dir/file ssh remote.box.com "cat /dir/file | gzip" | gunzip > dir/file ssh remote.box.com "cat /dir/file | bzip2" | bunzip2 > dir/file
 * 1) local to remote
 * 1) combined with gzip or bz2
 * 1) remote to local
 * 1) combined with gzip or bz2

Network with netcat
Transfers over netcat can have miniscule CPU needs, unlike transfers over ssh. However the data is transmitted without encryption and authentication.

Destination box: nc -l -p 2342 | tar -C /target/dir -xzf -

Source box: tar -cz /source/dir | nc Target_Box 2342

For further CPU use reduction, lzop can be used in place of the tar z option for much faster but less effective compression.

Destination box: nc -l -p 2342 | lzop -d | tar -C /target/dir -xf -

Source box: tar -c /source/dir | lzop | nc Target_Box 2342

Network with socat Same idea as Netcat.

Destination box: socat -U - tcp4-listen:2342 | tar x -C /target/dir

Source box: tar c /source/dir | socat -u - tcp4:Target_Box:2342

We can use a variety of compression methods in a general way: Destination box: socat -u - tcp4-listen:2342 | ${UNZIP} | tar x -C /target/dir

Source box: tar c /source/dir | ${ZIP} | socat -u - tcp4:Target_Box:2342

We then define ZIP as one of the following:

* cat * lzop * gzip * bzip2

and UNZIP as: * cat * lzop -d * gunzip * bunzip2

What's the difference?

time -p tar /usr/src/linux-2.6.3 | ${ZIP} | cat > /dev/null

* cat: 182MB, 1.1sec * lzop: 64MB, 4sec * gzip --fast: 51MB, 9sec * gzip: 41MB, 18sec * gzip --best: 41MB, 49sec * bzip2: 32MB, 134sec

This all depends on how fast your network connection is, and how fast each computer is. If your processors/disks are really slow and your network is really fast, cat alone works best. If your processors/disks are fast but your network is slow (e.g., dialup or slow wireless), you're better off with the smallest transfer (as would result from using bzip2 or gzip). Reality is, you're gonna be somewhere in the middle, but there's a reason that they put the kernel in .bz2 on kernel.org.