Peercache - Locally Shared Caches of Build Dependencies
The idea
Currently most build systems have two sources for build dependencies; A remotely hosted source (e.g. maven central, internal build cache on a server), or a local on-disk cache, but many development groups have another potential source which doesn’t seem to be being used at the moment; other developers on the same LAN, which is why I started thinking about implementing a peer-to-peer caching system and I’m putting the idea out there for discussion.
Many developers work with other, physically co-located, developers on fast LANs (either WiFi or wired), all of which share a much slower connection out to the internet or a server farm. When a dependency changes in an app or a library they’re working with all of the developers machines will go over the slow link and fetch the files necessary to complete the build, meaning multiple fetches of the same thing over a slow link. There are existing ways to work around this, but it usually involves setting up a local HTTP proxy which probably won’t be able to cache HTTPS data, and so any dependencies linked to via the HTTPS protocol will still be fetched multiple times.
We already have a solution for distributing download load; BitTorrent, so what I’m looking at doing is use BitTorrent (or a similar protocol) to locally distribute any dependencies which another machine on the LAN has downloaded.
The details
The initial approach will use a network broadcast to see if any other machines on the local subnet have already downloaded all, or part, of the dependency. This will generate either no responses, in which case that lucky machine gets to be the first to download the dependency from the original source, or it will generate one or more responses containing the following information;
- The other machines “load” (i.e. its ability to serve requests)
- The number of chunks the dependency has been split into (using a predictable formula to ensure all clients split a dependency into the same number of chunks).
- The chunks which the responding machine can serve to its peers.
This triggers the following process on the client which needs to download the dependency;
- Sort the list of chunks by scarcity based on the responses it has received (i.e. find which chunks are on the fewest machines).
- For each chunk identify which of the responding machines has the lowest load.
- Start downloading a number of chunks in parallel from the machines with the lowest load for those chunks which have the lowest availability.
- If there are multiple chunks with the same scarcity value, prioritize downloading the chunks which can be served by a machine with the lowest load.
If the dependency is a large one, the client may rebroadcast the request at intervals during its download to update its list of chunk scarcity and peer load.
Beyond the thunderLAN
Due to the initial implementation being based on a network broadcast this will mean the ability of the system to work in certain environments will be limited (e.g. where client isolation is used, or where machines are split over multiple separate subnets). A future extension could introduce the use of something like the tracker part of the BitTorrent protocol which would provide a synchronizing server to allow peers to communicate between different LANs.
Credits
This post was sparked by a “down the pub” conversation with Simon Stewart.