The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.
FLOP:SquashFS sync
Funtoo Linux Optimization Proposal: SquashFS sync
Problems with the current syncing method
Currently the official syncing method for Funtoo systems is git. While it's quite good, it has a few downsides:
- it places a lot of small files on the underlying filesystem — usually requiring a dedicated, small file-efficient filesystem,
- the git metadata increases disk space use even further, especially after multiple syncs,
- rare syncs imply transferring intermediate changes that may no longer be useful to end user.
While this is acceptable for hackers, it may be undesired for stable systems that are updated rather rarely and should be easy to set up.
Suggested system
Procedures
Funtoo provides daily repository snapshots as SquashFS images. Aside to complete today's snapshot, it provides one or more deltas from n day-old snapshot to the today's snapshot. All of those files are provided via HTTP(S).
User fetches the today's snapshot on first sync. Afterwards, if he updates within the supported m-day period, he only fetches a single delta and uses it to update the snapshot.
Advantages
The most important advantages of this system are:
- SquashFS allows to fit the whole repository in a single file, fitting any filesystem,
- SquashFS is both fast and space-efficient,
- deltas allow for fast and bandwidth-efficient updates.
Mirror impact
A single complete SquashFS image is around ~100 MiB large. A single day delta is around 10 MiB, and raises to ~30 MiB for 3-week period and 50 MiB for >3 months.