Created on: 2015/02/27
Original Author(s): Mgorny
Status
Reference Bug: FL-2138

Funtoo Linux Optimization Proposal: SquashFS sync

An alternate syncing method using SquashFS Images and deltas. Meant for stable systems where frequent updates aren't necessary, offering reduced bandwidth and disk space use, and improved speed.

Problems with the current syncing method

Currently the official syncing method for Funtoo systems is git. While it's quite good, it has a few downsides:

it places a lot of small files on the underlying filesystem — usually requiring a dedicated, small file-efficient filesystem,
the git metadata increases disk space use even further, especially after multiple syncs,
rare syncs imply transferring intermediate changes that may no longer be useful to end user.

While this is acceptable for hackers, it may be undesired for stable systems that are updated rather rarely and should be easy to set up.

Suggested system

Procedures

Funtoo provides daily repository snapshots as SquashFS images. Aside to complete today's snapshot, it provides one or more deltas from n day-old snapshot to the today's snapshot. All of those files are provided via HTTP(S).

User fetches the today's snapshot on first sync. Afterwards, if he updates within the supported m-day period, he only fetches a single delta and uses it to update the snapshot.

Advantages

The most important advantages of this system are:

SquashFS allows to fit the whole repository in a single file, fitting any filesystem,
SquashFS is both fast and space-efficient,
deltas allow for fast and bandwidth-efficient updates.

Mirror impact

A single complete SquashFS image is around ~100 MiB large. A single day delta is around 10 MiB, and raises to ~30 MiB for 3-week period and 50 MiB for >3 months.

FLOP:SquashFS sync