The Evolution of Unix’s Filesystem: From Necessity to Compatibility

Prehistory unixu

Some things in our world came about out of necessity, oversight, or pure chance. It was just like that here, but we really have to start from the beginning. When gentlemen Ken Thompson a Dennis Ritchie were creating the first Unix operating system in the late 1960s and early 1970s. They started on a PDP-7 computer and then ported the system to the more advanced PDP-11 in 1971.

This machine had two disk storages, each allowing storage 1.5 MB that is. The operating system was of course stored on one of them and the root file system gradually grew. So it happened that the first disk started to run out of space.

That’s why the two creators decided to start using the second disk as well and moved to it user data. They then attached the contents of the second file system to the directory /usr as a user, i.e. a user. There they set up subdirectories with the same names they used on the first disk: /bin, /sbin, /lib and more. Further development work continued there.

Later, they got a third drive for their computer and decided to make another change. They started a directory /home, they connected new capacity to it and moved user data there again. At that moment only happened department of user space in which data does not mix with executable programs in the operating system.

Where with him?

However, this created a new problem: where are we going to put the new files? What comes to the original root drive and what to slide into /usr? He already had it completely another function, did not contain user data, but was still named “user”. But it was actually a kind of extended storage for the operating system, so it was necessary to come up with some rules for it.

The framework was quite clear and unambiguous. The system boots from the master file system and must be able to mount a second file system at some point /usr. Essential files important for boot so they must not be in this extended repository. Command mount for example, you must not put there, otherwise you will never complete the start of the system and end up with they look bad without glasses.

Later, different ones were invented artificial rules and the criteria according to which this belongs here and that elsewhere. Unofficially, it is said that the “essential” system files belong on the main drive and the “rest” on the second /usr. But this separation was never strictly observed, so it gradually became diluted to the point where it ceased to make any sense.

This directory schizophrenia /bin a /usr/bin so it was created decades ago as a kind of implementation detail and is maintained by people who never asked why we actually do such a thing. It’s a classic pattern that we know from many other fields: we learn something, we continue to do it, and we end up doing it out of inertia, even though the world around us has changed and it doesn’t make sense.

It is no longer needed

The need for a separate file system to start an operating system actually disappeared long before Linux was created. First of all, it was created boot ramdisk (initrd and initramfs), which is loaded into memory in the early stages of startup and contains everything needed to start the rest of the system. So we don’t need to have another temporary boot filesystem on disk.

They threw the proverbial pitchfork into the Unix disk partition shared libraries and their introduction in the mid-1980s. Note that we are still a long way from any Linux. The concept of shared libraries does not allow the contents of directories to be maintained and operated separately /lib a /usr/bin. Their content must be in sync or it won’t work. Before the introduction of shared libraries, this was not a problem because all binaries were built statically.

Of course, too discs grew, so the original problem with the location also disappeared. In the 1990s, we were in the hundreds of megabytes, at the turn of the millennium in gigabytes, and today in terabytes. Today’s Unix operating system takes up a fraction of the capacity of a single disk, and much more space is needed for user data. We no longer need to separate part of the system on another disk.

Other rules

In the course of this convoluted history, Linux was born, which made the division of the file system simple took over and continued in it. Because that’s how it’s always been done. On top of that, it was created Filesystem Hierarchy Standardwhich formalized the whole situation and actually anchored it even more firmly.

However, we still have individual distributions, i.e. specific operating systems developed by specific groups of people. They add their own to the already established rules, so the whole thing is still happening more complicated. For example, the contents of a directory /tmp is deleted on reboot, but /usr/tmp no longer. But some distributions /usr/tmp they don’t have it at all and some have it as a link to /var/tmp. Because they are trying to follow another rule that the directory /usr is meant to be read whereas /var on the contrary, it is strongly writeable.

Then it may occur to you that even by itself / it’s actually read-only. That is, with the exception of the form /etcso from a bureaucratic point of view it would actually make sense to link /etc do /var. This way we could go on and on and pile up more wild rules complicating traffic. In practice, however there is no reasonthe important thing to remember is that the original division of the world came about only because someone once ran out of disk space.

More compatible with the rest of the world

Linux distributions started a unifying process some time ago. The goal is to get rid of separate content in directories /bin, /sbin, /lib a /lib64 and everything unite v /usr. This will eliminate a lot of artificial rules, but also make Linux distributions more compatible with the rest of the world.

Scripts and programs written for other Unixes will not need to be modified for specific paths in one Linux distribution or another. Roads /usr/bin a /bin they will be equal and there will be no need to solve which files we find in which of them.

The compatibility of the distributions with each other will also be better. Most Linux software is built using GNU autoconf/automake from GNU autotools, which does not handle partitions /usr specific to Linux. Preservation of division /usr requires non-trivial treatment project-specific in the build system and in the packages of each distribution. With the merger /usr this work becomes unnecessary and porting packages to Linux will be easier.

You may have thought that it would be more logical to get rid of the directory instead /usrwhich was created later, and keep only the original and simple /bin and its peers in the root directory. The problem of this solution is again given by historical custom. A number of current build systems are broken and doesn’t count with the possibility that the prefix for the installation path would be set simply to / or an empty string.

Sometimes it goes fast, sometimes slower

This whole story is actually just a kind of historical interest with an overlap into the present. Actually, it’s quite banal technical change, which involves transferring some files between directories. Distributions didn’t have much of a problem with that, things just gradually changed. Most users probably didn’t notice anything, because everything works for them as before.

It’s still a topic because not all distributions had a smooth transition. For example Debian it’s still struggling with it, and even the upcoming version 12, codenamed Bookworm, doesn’t have the directory merge done. Some distributions choose to do everything at once, during the time between releases. One version took care of it, and another already merged the directories. Debian is more careful and has spread the necessary steps over several versions.

If you look closely at a fresh installation of Debian 11 Bullseye or the upcoming version 12 “Bookworm”, you will find that directories like /bin a /lib they exist only as symbolic links to equivalent directories in /usr. So it seems that the merge process is complete and there is nothing more to deal with. Unfortunately, there are all sorts of sharp edges that need to be cooled.

Most of the problems are due to the fact that much of the transition was performed outside dpkg. This results in a situation where the package manager does not have a full understanding of the file system structure. Symbolic links at the top level of the file system structure thus create multiple paths to files that the packaging system does not understand. If files are renamed and moved between packages, the packaging system may delete files belonging to another package. This will lead to system damage.

Similar problems are why dpkg lists on merged systems /usr nasty warnings. The entire transition involves moving the files to their new locations. For example, a Bash package is modified to include a binary instead of a do /bin placed newly in /usr/bin. But there are a lot of such packages, and when you add to that the fact that moving files between packages is a fairly common thing, the chance of an error grows to unpleasant proportions.

Users definitely don’t want damaged systems after an update, and Debian doesn’t want to allow anything like that. Therefore, this error is a significant problem. It is also the reason why he maintains the project moratorium to move files to their new locations.

But there are far more problems, for example, when installing the system during bootstrapping, the whole process fails, because the packages during the processing of the installation scripts links to interpreters are missing. Under normal circumstances, they arise up to in the final phase installation, but with boostrap we need them right at the beginning.

The developers are of course debating over various proposed solutions a adjustments to the assembly process, which would alleviate the difficulties. It eventually succeeds, but it shows how difficult and time-consuming it can sometimes be to solve a problem that has arisen by chance fifty years ago. But half a century is a long enough time for even such a banality to take root quite firmly in our lives.