Shared Composer Vendors: beating the inode limit on shared hosting

shared-composer-vendors is a single PHP file that replaces composer install in a deploy script. It keeps one copy of each package version in a shared folder beside your projects, and turns each project's vendor/ into a few hundred links pointing at it. Ten Laravel sites on the same version of the framework stop holding ten copies of it, and the hosting account they share stops running out of files.

This is written for PHP developers who host several Composer projects on one account, on shared hosting especially. If each of your sites runs on a server of its own, you do not need it.

The story behind it

I wrote the first version as CTO of a software company delivering client projects on a monthly cadence. The projects were Laravel, and they were hosted side by side on one hosting account. What that account ran out of first was not disk space or memory. It was inodes.

An inode is the entry a Linux filesystem keeps for every file and every folder. Shared hosting plans cap how many an account may hold, and the cap sits on the plan page next to the disk space, where almost nobody reads it.

Composer is what takes a PHP project there. A Laravel project's vendor/ holds between 13,000 and 20,000 files and folders, around 150 MB, and it is by far the largest part of the project. Ten projects on nearly the same packages spend more than 150,000 inodes and 1.5 GB on copies that are almost identical: the same laravel/framework, the same Symfony components, the same Carbon, once per site.

Reaching the cap does not slow anything down. The host simply refuses new files, so an upload fails, a cache cannot be written, and the next composer install stops halfway through. The usual fix is the next hosting plan up, and with a new project every month we were heading there on a schedule.

The first version of this tool, a bash script with a PowerShell twin, cut the account's inode usage by 40% and put that upgrade off by years. The version I am publishing is a rewrite in one PHP file, which runs the same on Linux and Windows. Before releasing it I built a Laravel project whose only job is to break it: a package patched on install, a Composer plugin that edits its own files, a package installed from git, another from a local folder, and mPDF writing temporary files into its own folder.

One copy per version, one link per project

The idea fits in one picture:

# Without the tool: every project holds its own copy
site-a/vendor/laravel/framework/     # 1,600 files
site-b/vendor/laravel/framework/     # 1,600 files

# With the tool: each project holds a link, and the files exist once
site-a/vendor/laravel/framework  ->  shared-vendors/vendors/laravel/framework_13.33.0/
site-b/vendor/laravel/framework  ->  shared-vendors/vendors/laravel/framework_13.33.0/

A link is a tiny entry that points to a folder somewhere else, and PHP reads through it as if the files were really there. On Linux it is a symbolic link. On Windows it is a junction, which needs no administrator rights. Each shared copy is stored under its version, so a project still on an older release links to that one, and two versions of a package live side by side for as long as a project uses each of them.

Each project's vendor/ shrinks to a few hundred links, and a new site built on packages already in the shared folder costs almost no inodes at all.

Installing shared-composer-vendors

It needs PHP 8.0 or later and Composer 2, with Composer's download cache left on, which is the default. Clone it into the folder above your projects. The shared folder is created inside it on the first run.

cd ~
git clone https://github.com/mehdismekouar/shared-composer-vendors.git shared-vendors

Then, in each project's deploy script, replace composer install with a call to setup.php:

cd ~/domains/site-a
git pull

# Before: composer install --no-dev
php ~/shared-vendors/setup.php

php artisan migrate --force

Like composer install --no-dev, it leaves development packages out unless you pass --dev. A few options cover the rest:

• --dry previews what would happen without changing anything, in the project or in the shared folder.

• --wait SECONDS sets how long a deploy waits for its turn, since only one project deploys at a time. The default is ten minutes.

• --composer "php ~/composer.phar" tells it how to start Composer on a server where composer alone does not work.

• Anything after -- is passed on to composer install, for example -- --optimize-autoloader.

It exits with 0 when the site is deployed, and with 1 when the deploy failed and either nothing was changed or everything was put back. The rest of your deploy script can check it like any other command.

What happens during a deploy

A deploy goes through five steps, and the site keeps running through all of them.

It builds the new vendor/ beside the live one. composer install runs into a folder called vendor.next, while the site goes on using the current vendor/.

It swaps the two by renaming them. The live vendor/ becomes vendor.old and vendor.next becomes vendor/. A rename is instant, so no request ever sees a half-installed vendor/, and if anything failed before this point nothing has changed.

It runs your Composer scripts, Laravel's package:discover among them, at the same moments a normal install would. If one fails, the previous vendor/ is put back.

It shares the packages, one at a time. Each package is replaced by its link with two more renames, so it is never missing for more than an instant.

It cleans up. The previous vendor/ is deleted and a summary is printed.

Which packages get shared, and how it knows

The rule is strict on purpose: a package is shared only when it is exactly what Composer downloaded.

Composer keeps the zip of every package it downloads in its cache folder, and a zip already stores a checksum for every file inside it, in an index at the end of the archive. So the tool reads that index, computes the same checksum for the installed files and compares the two lists, without unzipping anything. If they match, the package is linked to the shared copy, or becomes the shared copy if this project is the first one on that version.

A package can differ from its download for good reasons: a patch applied by composer-patches, a Composer plugin writing its own configuration, a script of yours editing a file after install. Those stay in the project, with the reason printed on their line, until a second project makes exactly the same change. The changed version is then stored under a name of its own, and both projects link to it.

Waiting for that second project is deliberate. With only one project, sharing a changed copy saves nothing. And some changes are different on every deploy, a generated file with a date in it for instance, so storing each one would add a full copy of the package to the shared folder every time you deploy.

Two kinds of package are never shared: one installed from a git repository or a local folder, whose files can change without its version changing, and one you list yourself in composer.json. Wildcards work:

"extra": {
    "non-shared-vendors": ["some/package", "some-vendor/*"]
}

The packages that do not like being a link

This is where most of the work went. When PHP works out __DIR__ and __FILE__, it follows links, so a file reached through a link knows its real address in the shared folder rather than its address in your project. Most packages never notice. Two kinds do.

Tools that find the autoloader by climbing folders. The command-line tools in vendor/bin, php-parse or var-dump-server for example, look for vendor/autoload.php a fixed number of folders above themselves. Counted from the shared folder, three folders up is the wrong place, and the tool fails.

Packages that write into their own folder. mPDF writes temporary files into its own tmp/ folder while your site runs. The shared folder is read-only, so those writes would fail, and if it were not, every project would be writing into the same folder.

For those packages the project gets a real folder instead of a single link. The files that look around sit at the project's own address as hard links, a second name for the same file that costs no space and no extra inode. The folders the package writes into are real, writable and private to the project. Everything else still points to the shared copy. For mPDF it looks like this:

vendor/mpdf/mpdf/           # a real folder
├── data/, ttfonts/         # links to the shared copy: most of the package
├── src/Config/...          # real folders, hard-linked files: they decide where tmp/ is
└── tmp/                    # a real, writable folder that belongs to this project

The package works as usual, for a handful of extra files, usually between 3 and 20. The tool finds what needs this by reading each package's code once, when a version first enters the shared folder: a __DIR__ path that climbs above the package, a file_put_contents or an fopen in write mode aimed at its own folder. When it finds a write whose target is only known while the site runs, it cannot place it, so it warns you instead, and the fix is either to point that package's cache folder at your project's storage/ or to list it in non-shared-vendors. In my test project, out of 90 packages, mPDF is the only one that needs a writable folder of its own.

Why the shared folder is read-only

I weighed making it writable, which would have made mPDF and its kind trivial. What decided it is that a writable shared folder turns every mistake into a mistake on every site at once: a package deleted through a link by someone cleaning up one project, a quick edit made on the server to debug one site, malware that got in through any of them.

Read-only, one project cannot change what the others run, and the few packages that need to write get a folder of their own. A new version is also copied in under a temporary name and renamed only once it is complete, so a deploy that dies halfway never leaves a half-copied package behind for the next project to link to.

Reading the output

Every package gets one line, and the line says why when a package is not shared:

[4/5] Linking packages to the shared folder
  [LINK]  laravel/framework v13.33.0
  [NEW]   filament/filament v3.3.43
  [LINK]  mpdf/mpdf v8.3.1 - has its own writable folder in this project: tmp
  [LOCAL] psr/log 3.0.2 - patched by composer-patches (changed: src/LoggerInterface.php); it will be shared as soon as another project makes the exact same change
  [LOCAL] ramsey/collection 2.1.1 - installed from a git repository, so its files can change without a new version

[LINK] means the package was already in the shared folder, [NEW] that this deploy added it, and [LOCAL] that the project keeps its own copy. A package that could not be linked shows as [FAIL] in red and keeps its own copy too, but the deploy still counts as a success: the site works, and the migrations after it in your deploy script still need to run.

A summary closes the run. On my test project, a Laravel 13 application with 90 production packages, a repeat deploy links 85 of them and shares 7,247 files instead of copying them. The other five keep their own copies, each for the reason printed on its line.

Limits worth knowing

• For about two minutes after a deploy that changes a package's version, some requests may still read the old version's files. PHP remembers where each path leads for 120 seconds, its realpath cache, and the old version is still in the shared folder. If your host lets you restart PHP, do it after deploying.

• Old versions are never deleted. When no project uses a version any more, it stays in the shared folder until you remove it by hand, and the README shows how to check that nothing links to it first.

• Composer's download cache has to stay on disk between deploys, since it is what packages are compared against. A package whose zip is gone from the cache is shared only if it matches the copy already in the shared folder.

• Every deploy installs into a fresh folder, so Composer only ever sees installs, and scripts on package update or uninstall events never run. Put that work in post-package-install.

• Windows works, with junctions instead of symbolic links and copies instead of hard links, but it has been tested on a development machine and not yet on a Windows production server. One trap there: in Git Bash, rm -rf on a linked package with a trailing slash, the one Tab completion adds, deletes the shared copy for every project.

Where to get it

The tool is on GitHub, under the MIT licence.

If you run several Laravel sites on one hosting account and want the deploys behind them set up to last, that is the work I do.