How to combine/merge 2 or more folders virtually(sym/hardlinks):
#################################################################
#################################################################
URL: http://unix.stackexchange.com/questions/40634/in-ubuntu-is-there-a-way-to-virtually-merge-two-folders-without-unionfs-or-aufs
Lets say you have a Movies1 and Movies2 folder and you want their contents to be visible through a Movies folder (which has the combination of Movies1 and Movies2). They dont take up the extra space thus its virtual (because symlinks/hardlinks are involved)
Or more example type of scenario imagine folder 1 and 32 that you want to combine to folder 3:
START:
——
/testfolder
├── 1
│ ├── s
│ ├── x
│ ├── y
│ └── z
│ └── 1
└── 2
├── a
│ └── 2
├── b
└── c
FINISH:
——-
/testfolder
├── 3
├── s
├── x
├── y
└── z
| └── 1
|
├── a
│ └── 2
├── b
└── c
Note: 1 and 2 still exist they are just not shown
SIDE NOTES:
———–
* 3 is not in alphabetical order but it would be after the combine with a simple “tree” command.
* To get similar output download “apt-get install tree”
* shopt -s dotglob is used below for * to also “grab” hidden files
* Source file paths have to be absolute in such case (that’s why I used $PWD), because cp “can make relative symbolic links only in current directory”.
SOLUTION:
———-
You can “copy” whole tree using symlinks (or hard links if you want) automatically using -s option of cp (or -l for hard links).
# cd /testfolder
# mkdir 3
# shopt -s dotglob # <—- optional – turns on dotglob to grab hidden dot files with *
# cp -ans “$PWD/1/”* 3/
# cp -ans “$PWD/2/”* 3/
# shopt -u dotglob # <—- optional – turns off dotglob if you turned it on
VARIATION NOTE: the ” ” are optional so you could just put:
# cp -ans $PWD/1/* 3/
# cp -ans $PWD/2/* 3/
VARIATION NOTE: instead of $PWD/1/* and $PWD/2/* – I could just use the full path of the folders with an * ending.
mkdir /testfolder/3
shopt -s dotglob
cp -ans /testfolder/1/* /testfolder/3/
cp -ans /testfolder/2/* /testfolder/3/
shopt -u dotglob
NOTE ON SPEED: This is a very quick process and can be monitored -v (for verbosity) shows every transfer – realize that it doesnt really transfer it just makes a link so it makes a small (very small file) so this process is much faster then a regular cp
NOTE: $PWD is just a variable kept in the bash system at all times which tells bash what current folder your in – random fact: changing that variable changes your path – to see what it equals just type # pwd (which is a command that simple does the following # echo $PWD) – infact try it “echo $PWD” that will list the current directory
ANOTHER INTERESTING SOLUTION:
——————————
* You can also do the case of having folder 1 to 2 where there is no folder 3, the end result would feel the same. So folder 2 would have symlinks/hardlinks to folder 1. A user in folder 2 would see folder 1 and folder 2 contents
# cd /testfolder
# mkdir 3
# shopt -s dotglob # <—- optional – turns on dotglob to grab hidden dot files with *
# cp -ans “$PWD/1/”* 2/
# shopt -u dotglob # <—- optional – turns off dotglob if you turned it on
END RESULT:
/testfolder
├── 2
├── s
├── x
├── y
└── z
| └── 1
|
├── a
│ └── 2
├── b
└── c
Note: 1 still exist its just not shown
Explanation of used cp options:
——————————–
-a or –archive
preserves attributes, links and copies directories recursively (it’s in fact alias of -dR –preserve=all)
-n or –no-clobber
prevents overwriting existing files
-s or –symbolic-link
makes symbolic links instead of literal copying
-l to make hard links
PART 2 – Unvirtualizing the result folder (or simply dereferencing the links)
##############################################################################
Lets say we want to convert all of the symlinks and hardlinks to originals, first lets look at why.
So after the copy with -ans (or -vans for more verbosity with the v) – thats the -ans copy from above – we have something that looks like this:
/testfolder
├── 3
├── s
├── x
├── y
└── z
| └── 1
|
├── a
│ └── 2
├── b
└── c
Which is a combination of 1 and 2
However in reality everything within 3 is just symlinks or hardlinks to the actual data so it all looks like this actually
/testfolder
├── 3
├── s -> /testfolder/1/s
├── x -> /testfolder/1/x
├── y -> /testfolder/1/y
└── z
| └── 1 -> /testfolder/1/z
|
├── a
│ └── 2 -> /testfolder/2/a/2
├── b -> /testfolder/2/b
└── c -> /testfolder/2/c
Lets say you wanted to “unvirtualizing” them so it actually looks like this:
/testfolder
├── 3 <—————— this will be 4
├── s
├── x
├── y
└── z
| └── 1
|
├── a
│ └── 2
├── b
└── c
Well in reality I dont know about dereferencing each link within 3, however we can make a new folder – lets say /testfolder/4 – where each file will be a none link file (its called copy with dereferencing). So it will be like this
/testfolder
├── 4
├── s
├── x
├── y
└── z
| └── 1
|
├── a
│ └── 2
├── b
└── c
HOW TO:
# cd /testfolder
# mkdir 4
# shopt -s dotglob
# cp -RL 3/* 4
# shopt -u dotglob
NOTE: Dereferencing does a full copy instead of making a small link so this process is much longer
NOTE: again the dotglob is optional – its just to grab the hidden dot files. So if they dont exist in 3 then no need to do the dotglob commands.
SECTION 3 – REAL WORLD EXAMPLE OF SECTION 1 AND SECTION 2 WITH APT-MIRROR AND REPOSITORIES
###########################################################################################
Example of use:
apt-mirror is a program that downloads a full repository to your ubuntu/debian box (you feed it a configuration file which looks similar to sources.list from apt, and it downloads the full repositories).
apt-mirror saves everything to the save location (namely the base_path variable). My base_path is /Sally/Studies/repo1.
The actual pacakges save into
/Sally/Studies/repo1/mirror/{domain name of repo}
The metadata of the packages which is equaly important for the repository to work is saved into
/Sally/Studies/repo1/skel/{domain name of repo}
So I was downloading the ubuntu and the debian repository so this is what I had after a successful apt-mirror:
/Sally/Studies/repo1/mirror/gb.archive.ubuntu.com/..stuff..
/Sally/Studies/repo1/mirror/ftp.us.debian.org/..stuff..
/Sally/Studies/repo1/skel/gb.archive.ubuntu.com/..stuff..
/Sally/Studies/repo1/skel/ftp.us.debian.org/..stuff..
Problem with this split is that programs like debootstrap and apt-get need everything from skel and mirror to be together (or merged) in fact that abstraction only exists with apt-mirror (as far as I am aware – and I could be completely wrong)
So to combine them, first I pick a name for the combination folder – combo1. Nice and original
cd /Sally/Studies/repo1/
mkdir combo1
shopt -s dotglob
cp -vans /Sally/Studies/repo1/mirror/* combo1
cp -vans /Sally/Studies/repo1/skel/* combo1
shopt -u dotglob
Or another alternative as you see in section 1:
cd /Sally/Studies/repo1/
mkdir combo1
shopt -s dotglob
cp -vans $PWD/mirror/* combo1
cp -vans $PWD/skel/* combo1
shopt -u dotglob
Since the cp commands will take a minute you can do it all in one shot:
cd /Sally/Studies/repo1/
mkdir combo1
shopt -s dotglob
(cp -vans $PWD/mirror/* combo1 && cp -vans $PWD/skel/* combo1) | tee cp.log
shopt -u dotglob
Or instead of $PWD we can just use the full paths:
mkdir /Sally/Studies/repo1/combo1
shopt -s dotglob
(cp -vans /Sally/Studies/repo1/mirror/* /Sally/Studies/repo1/combo1 && cp -vans /Sally/Studies/repo1/skel/* /Sally/Studies/repo1/combo1) | tee /Sally/Studies/repo1/cp.log
shopt -u dotglob
So the final result is
/Sally/Studies/repo1/combo1/gb.archive.ubuntu.com/..stuff of skel and mirror combine – but just symlink versions of it..
/Sally/Studies/repo1/combo1/ftp.us.debian.org/..stuff of skel and mirror combine – but just symlink versions of it..
Now I can point an http server (like an apache2 server) to this folder – Like this:
ln -s /Sally/Studies/repo1/combo1/ /var/www/repos/
Now I can point to them with a remote machines (which still in the same lan – or even across the internet If I have my webserver and firewall correctly setup) with /etc/apt/sources.list
deb https://10.10.10.10/repos/ftp.us.debian.org/debian wheezy main
deb https://10.10.10.10/repos/ftp.us.debian.org/gb.archive.ubuntu.com precise main
Finally lets say I want a USB copy – I cant have it just be symlinks on the USB, I need the actual data – So I need the section 2 material to derefence the links:
Lets pretend sdX1 is our imaginary USB drive
# mount /dev/sdX1 /mnt
# mkdir /mnt/repocombo/
# shopt -s dotglob
# cp -vRL /Sally/Studies/repo1/combo1/* /mnt/repocombo
# shopt -u dotglob
NOTE: Dereferencing does a full copy instead of making a small link so this process is much longer