|
####################
TAR PV NC fileserver
####################
Pre Requisites:
At File Server Router: Portforwards port 50002 from router to 50002 on File Server
At both locations: apt-get install netcat tar pv
Note: File Server will be the destination, and the client will be the source
CLIENT 2 DESTINATION NO COMPRESSION
###################################
===File Server @ www.home.com:===
# while true; do nc -l -p 50002 | pv | tar -xf -; done
The while loop is for continuity, if the transfer at client stops, the file server nc command stops, but with the while loop it just restarts.
===Client @ wherever:===
–Simple – just shows transfer rate:–
# tar -cf – /srcfolder | pv | nc www.home.com 50002
–Complex – shows Time to Complete (need to specify size of traffic in bytes):–
Can get size of traffic in bytes like this: du -sb /srcfolder | awk ‘{print $1}’
-sb summarize the size of /srcfolder folder in bytes, and the awk part just grabs the bytes number part of the output and not the /srcfolder part
So combine it with the script using the -s arg from pv, it needs size in bytes by the way:
# tar -cf – /srcfolder | pv -s `du -sb /srcfolder | awk ‘{print $1}’` | nc www.home.com 50002
CLIENT 2 DESTINATION WITH COMPRESSION
#####################################
===File Server @ www.home.com:===
# while true; do nc -l -p 50002 | pv | tar -xzf -; done
The while loop is for continuity, if the transfer at client stops, the file server nc command stops, but with the while loop it just restarts.
===Client @ wherever:===
–Simple – just shows transfer rate:–
# tar -czf – /srcfolder | pv | nc www.home.com 50002
–Complex – shows Time to Complete (need to specify size of traffic in bytes):–
Can get size of traffic in bytes like this: du -sb /srcfolder | awk ‘{print $1}’
-sb summarize the size of /srcfolder folder in bytes, and the awk part just grabs the bytes number part of the output and not the /srcfolder part
So combine it with the script using the -s arg from pv, it needs size in bytes by the way:
# tar -czf – /srcfolder | pv -s `du -sb /srcfolder | awk ‘{print $1}’` | nc www.home.com 50002
—————————————
Test 1: My speeds were 600 kB/s without Compression
Test 2: With Compression and decompression same speed … maybe like 50 kB/s faster
—————————————
##############################
TAR PV SSH Copy Folder or File
##############################
This builds on the idea that you can transfer a file with ssh using a command like this:
cat filein | ssh destination.com -c arcfour,blowfish-cbc -C -p 50005 “cat – > /tmp/fileout”
Where filein is the source file, destination.com is the ssh server im copying the file to, -c arcfour,blowfish-cbc is selecting the fastest encryption engines, -C is for online compressions and decompression when it comes off the line – supposed to speed up tx in some cases, then the /tmp/fileout is how the file is saved…
So im taking that method along with tar piping and pv piping to provide you with a way to ssh full directories or filesystems with a progress bar 🙂
Benefits over the nc way is that this is done with one command from one side, there is no need to have a listening nc running on the server 🙂 The idea is openssh should already be listening its the way openssh works in linux 🙂 This does require a password to authenticate in, of course if you have the key auth. setup the right way then a password will not be required, but typically yes you will have to type in a password.
NOTE: When doing these commands when asked for questions there might be flowing text from the pv doing the progress bar just continue typing as if its not there, close your eyes if it helps, there might be a yes or no question, type “yes” and ENTER to it, and also it will ask for a password, just put in your password and ENTER
Pre Requisites:
At File Server Router: Portforward some port, 50005 in this case, to port 22(or your ssh port) on your ssh server, this should be a given already since your this far ahead in your linux studies
At both locations: apt-get install openssh tar pv
At Server (destination) make sure your ssh server is operating, test by ssh localhost from it. Again this is also a given if your already this far ahead in your linux studies.
—WITH COMPRESSION/DECOMPRESSION:—
VIA SSH WITHOUT PV:
# tar -czf – /srcfolder | ssh -p 50005 root@destination.com “tar -xzvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar:
# tar -czf – /srcfolder | pv -s `du -sb /srcfolder | awk ‘{print $1}’` | ssh -p 50005 root@destination.com “tar -xzvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar & FASTEST SSH ENCRYPTION:
# tar -czf – /srcfolder | pv -s `du -sb /srcfolder | awk ‘{print $1}’` | ssh -c arcfour,blowfish-cbc -p 50005 root@destination.com “tar -xzvf – -C /dstfolder”
NOTE: Can implement -C on the ssh however, there will be no benefit logically speaking since we already did all the possible compressions at the tar level
NOTE: Also note that this way the destination will have the following folder structure /dstfolder/srcfolder
—WITHOUT COMPRESSIONS/DECOMPRESSION:—
VIA SSH WITHOUT PV:
# tar -cf – /srcfolder | ssh -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar:
# tar -cf – /srcfolder | pv -s `du -sb /srcfolder | awk ‘{print $1}’` | ssh -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar & FASTEST SSH ENCRYPTION:
# tar -cf – /srcfolder | pv -s `du -sb /srcfolder | awk ‘{print $1}’` | ssh -c arcfour,blowfish-cbc -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
NOTE: Also note that this way the destination will have the following folder structure /dstfolder/srcfolder
—WITHOUT COMPRESSIONS/DECOMPRESSION FROM SSH:—
VIA SSH WITHOUT PV:
# tar -cf – /srcfolder | ssh -C -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar:
# tar -cf – /srcfolder | pv -s `du -sb /srcfolder | awk ‘{print $1}’` | ssh -C -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar & FASTEST SSH ENCRYPTION:
# tar -cf – /srcfolder | pv -s `du -sb /srcfolder | awk ‘{print $1}’` | ssh -C -c arcfour,blowfish-cbc -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
NOTE: Also note that this way the destination will have the following folder structure /dstfolder/srcfolder
=======================================================
=======================================================
=======================================================
=======================================================
Dont like? /srcfolder to /dstfolder/srcfolder
Lets say you dont want the /dstfolder/srcfolder stuff, which is understandable im also OCD about filestructure, I cant stand fixing it afterwards, so this is how to do it this way
Want it to be like? /srcfolder to /dstfolder, without the srcfolder folder nested inside dstfolder…
If this made sense here are the above commands redone for this…
How do I do this, whats different from above? Not much, I just change all the /srcfolder to ., and you have to make sure to cd to /srcfolder before running the commands…
—WITH COMPRESSION/DECOMPRESSION:—
VIA SSH WITHOUT PV:
# cd /srcfolder; tar -czf – . | ssh -p 50005 root@destination.com “tar -xzvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar:
# cd /srcfolder; tar -czf – . | pv -s `du -sb . | awk ‘{print $1}’` | ssh -p 50005 root@destination.com “tar -xzvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar & FASTEST SSH ENCRYPTION:
# cd /srcfolder; tar -czf – . | pv -s `du -sb . | awk ‘{print $1}’` | ssh -c arcfour,blowfish-cbc -p 50005 root@destination.com “tar -xzvf – -C /dstfolder”
NOTE: Can implement -C on the ssh however, there will be no benefit logically speaking since we already did all the possible compressions at the tar level
NOTE: Also note that this way the destination will have the following folder structure /dstfolder
—WITHOUT COMPRESSIONS/DECOMPRESSION:—
VIA SSH WITHOUT PV:
# cd /srcfolder; tar -cf – . | ssh -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar:
# cd /srcfolder; tar -cf – . | pv -s `du -sb . | awk ‘{print $1}’` | ssh -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar & FASTEST SSH ENCRYPTION:
# cd /srcfolder; tar -cf – . | pv -s `du -sb . | awk ‘{print $1}’` | ssh -c arcfour,blowfish-cbc -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
NOTE: Also note that this way the destination will have the following folder structure /dstfolder
—WITHOUT COMPRESSIONS/DECOMPRESSION FROM SSH:—
VIA SSH WITHOUT PV:
# cd /srcfolder; tar -cf – . | ssh -C -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar:
# cd /srcfolder; tar -cf – . | pv -s `du -sb . | awk ‘{print $1}’` | ssh -C -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
VIA SSH WITH PV – gives progress bar & FASTEST SSH ENCRYPTION:
# cd /srcfolder; tar -cf – . | pv -s `du -sb . | awk ‘{print $1}’` | ssh -C -c arcfour,blowfish-cbc -p 50005 root@destination.com “tar -xvf – -C /dstfolder”
NOTE: Also note that this way the destination will have the following folder structure /dstfolder
|