Splitting and recombining larger files

Why is the md5sum used? If you are using a zip, there is a 32bit crc inside the file.

If the concern is successful reception, would it be possible to split the bandwidth used into 2 signal streams with a slower signaling rate? Then send a primary feed and then a delayed feed of the same content of the primary. This would allow users who missed a bit of the primary signal could receive the secondary and successfully recreate the full transmission.

Or maybe use 2 Radios + LNA + Antennas, to provide a second reception point. This might help with odd noise issues.

1 Like

Another option would be to bend the quantum (single file) rule and include a PAR1,2,3 file or something that supports error correction. That way many of the parts of the file could be reconstructed if signal loss was experienced.

What if you didn’t zip text files as they were transmitted? That way if loss or corruption occurred, the rest of the document would still be intact and readable. This would be more of the design of UDP or broadcasts where the transmission is best effort and signal loss is to be expected.

Going to toss this in here because it seems like the right place. I’m not ready to share all the gory details because I still have some work to do putting a demo together.

I have been playing with a lossy text compression idea that I think would work well for this low bandwidth application. The idea is that you can reduce the size of text files using two primary mechanisms: 1) removing non-critical information from the text, and 2) reduce the size of lookup tables that the data compression programs (ZIP, 7Zip, etc) use. Sort of a pre-compression step.

The first item is pretty easy. A simple example of this is to change all the text in a file to be lower case. When zipped, the resultant file is smaller because there is less data in the file. Software can apply capitalization where needed…even if some proper names are not capitalized who cares?

The second item (reducing the dictionary) requires a common dictionary (or maybe more than one) that gets downloaded one time to the CHIP and is used to uncompress the files sent down from the satellite.

My last work session this weekend I have a 34000 byte file (CNN news stories) being compressed to 12000 bytes on the device which would be 9000 bytes over the wire (assuming 7zip is the final compression…its not but whatever). This same 34000 byte file compresses to 13500 bytes with 7zip and no lossy text tricks being applied…the benefits are significant.

The downside to this is that one or more dictionaries need to be present on the device in order to perform the final decompression step. Since you would need some Javascript in the clients browser to decode the text files, you could show a friendly error message telling the user to wait for the dictionary to be downloaded. Future versions of the CHIP might just include a set of dictionaries so they don’t need to be downloaded at all.

Sorry this got a little rambling…but there is lot to try and explain.

1 Like

which md5? for the flash file zip? its not used.

cause it reduces agility and breaks file integrity.

image updates are currently not happening, though they are coming. So issuing an image update for a minor styling fix would be ridiculous. In anycase, styling is a very small part of the whole thing, post compression. Of course, if anyone would like to optimize it further, I would love to see a PR.

it would be vastly simpler and cheaper to just use a larger Antenna. The small antenna is a feature.

Adding a second set won’t help - at all. If there is a local fade, won’t both sets fade? Also they would interfere with each other. There are ssomething called array antennae, and its something we are experimenting with.

I am not quite sure how that would help - the delayed feed is the same as the carousel going around again. Just slower, and userless for time sensitive content, and the bandwidth reserved for it is better used for more FEC.

zipping text files typically reduces their size by 5-7x. Not zipping them dramatically increases the chances of they not being transmitted fully. Zipping is a net win, just on that basis. But: broken files aren’t necessarily readable - depends on what it is, and how broken it is. broken quotes or end tag in html, would make it impossible to read even a text file. For other files, its not even an option.