Peer-to-peer bootstrapping a file collection?

One of the least exciting things about Outernet is that a receiver starts up “empty”, and there’s sort of a long wait for any useful amount of content to appear. You sort of have to walk away for a week in order to be surprised by how much stuff awaits upon your return. Even with a higher data rate in the future, it’s going to take a very long time to fill up even a small SD card, during which time that space is providing no value to anybody.

While that’s completely understandable, it’s also an opportunity of sorts. There are plenty of ideas for preloading content into that empty space, from Rachel to offline-Wikipedia to who knows what else.

But what if this was automatic? Suppose you’re a new Outernet user in a remote place. You may have heard of Outernet from someone who already has a setup. Suppose you visit them, and fire up your receiver right next to theirs for a little while, they find each other over wifi, and your box automatically mirrors theirs, to get some early content. Then when you take yours back to your place, it keeps itself updated over satellite, but it’s already chock-full-o-goodies from that initial transfer.

It was pointed out in this thread: We Moved! that “flash drives were notorious for picking up and spreading viruses in rural area deployments”, so that’s a concern here too.

It’s my understanding that data sent over the satellite is signed, and the signature is verified before unpacking. I believe the signature data is discarded at that point. But suppose it wasn’t? If the new node could verify signatures of the content it’s mirroring, it could hopefully have the same trust as stuff freshly received by satellite, and not be at risk of mirroring any stow-away malware.

This would suggest some changes in the way files are stored, perhaps keeping a substantial cache of files just as they were received from the satellite in addition to their unpacked versions, or perhaps keeping only the packed versions and accessing them “live” inside their compressed containers, in some FUSE-enabled abomination against all that is good and sane in filesystem design. Or whatever – I’m sure the comp-sci folks have better ideas than I.

Side note, this would also enable a node that was offline for a while to get the missed updates simply by being brought near another node that caught them. Which I consider a super minor feature but sort of nifty, and it’s actually the use-case that initially made me think about this whole scheme, after mine was down a while and I wondered what I missed.

Of course it would be way faster to simply do this with a USB-SD reader and swapping cards around, but if we imagine sealed boxes someday to dissuade the viruses scenario, wireless becomes important. And it’s just cool.

2 Likes

I’ve thought about such a thing as im running 4 separate outernet setups… when I install a new one its underwhelming…

So my idea was to have a “master” outernet system populating a storage site on the net where other newly installed systems could be “primed” with data via tethered internet connection. If the master system updates fast enough we should be able to have all the systems in sync rather quickly.

I built a simple script to run that uses rsync to make a remote connection and sync the downloads directory between two and it works fine.

I’m willing to host a source site on radionerds, but I dont have the best archive… anyone else have a long running system to prime the master with?

Chris

1 Like

I’m convinced that one can set up a Dreamcatcher with a USB Flash Drive (holding Rachel or whatever)) which will be protected by Skylark’s READ-ONLY protocol, and not have to worry about viruses. Ken

2 Likes

I do agree with you but if your using a DC and since the two sd card slots would give the same situation it would be easier to use them, and less likely to cause issues with physical damage.

Regardless it still works best if you can prime them before you put them online so its not empty on start.

Chris

1 Like

Chris, have been able to get the data SD slot to work? My DC won’t recognize it with or without files on it. I formatted a 16 GB SD chip FAT 32. Ken

1 Like

I do this on my network. I run an old style BBS. I Mount skylark/external/downloads/ to a local Mount point then sync . I employ no security or checking.