Uploading larger files?

Hi all

I realise it’s early days but I’m just wondering how it will work to upload larger files.

I realise that some stuff is time sensitive: News, weather etc. I guess this might use up half the available bandwidth? 10mb a day

Alongside that it would be good to have some ‘evergreen’ content such as this 53mb open-source chemistry textbook. These may be larger files and you wouldn’t want to prioritise them over the time sensitive stuff, so how will that work? Would they need to be split into chunks? Or does the carousel have software to handle different priority files?

Perhaps Outernet could sell an adaddendum set of DVD’s or put 64 gb of data on every CHIP etc as it is sold and this could consist of this HEAVY LIFT backbone data.

The 20 MB a day could then supplement this backbone data.

I would not vote for 2 and a half days data quota going to a 53mb chemistry book.

1 Like

I guess it could be compressed down a fair bit, but I guess my point was that it shouldn’t be two days of data it should be split over 30 days or so.

Hello Sam

I’m glad we’ve got other folks to comment on the content side of Outernet - - it seems to be a subject that has fallen by the wayside.

I have been experimenting with creating small files to upload, and have decided that PDFs work best at this time. The recent PDF with VOA news are my creation. I find the PDF render best on all the platforms with all the browsers I try. I can’t figure out how to open some of the file formats being transmitted. The .GZ self-extracting compressed formats need to be downloaded and opened separately, but ultimately look great.

When I create a PDF, I use a PDF printing program (www.doPDF.com) that installs as a standard printer on my computer. If the file is less than 100 kB - - I upload it. This program produces very small unshrinkable PDFs, so the PDF shrinking programs that are available on-line don’t make a difference.

If the PDF is larger, I have to reconstruct the PDF with my word processor and then save it out as a new PDF with lower resolution graphics to get it below 100 kB. Ken

1 Like

the .html.gz formats are meant to be a universally viewable solution - that includes mobile browsers.

They work out of the box on the full outernet reciever setup (the CHIP based one, for example). But yes, they are a pain to open on the desktop installers or OuternetInABox. We are working on easy viewability on the desktop setup.

PDFs are a problem - cross-linking between files doesn’t work, in our testing they were larger than the equivalent .html.gz, and mobile browsers have a very rough time with them.

1 Like

This one seems to work quite well https://online2pdf.com/pdf-reduce-size

It allows the compression of images, putting two pages to a sheet and got my above 400 page file down to 3Mb in colour at one page a sheet.

Or 2.5Mb at it’s smallest in black and white

I’ll try that on my next big PDF and let you know how it works out here.

Thanks, Ken

Follow-up. I tried it and got a 70 % reduction in size which is fantastic. I selected all low resolution settings, and rendered it in black and white.

The file I tried it on was 799 kB and it got reduced to 239 kB - - still too large to Filecast.

Interestingly enough, if I try it on one of my compressed PDF files that I’m creating, I get no improvement. I must have already done all the cleaning :innocent:
Ken

This is an example of a file that would be worth spending bandwidth on IMHO: It’s a medical guide for what to to Where there is no doctor it’s 3.7Mb it can be distributed not-for-profit and could really save lives.

1 Like

@sam_uk Agreed. We need to make a lower-sized version of that book. It will almost always be more efficient to compress text and include images, rather than sending a pdf.

PDFs are a minefield really: this one was fascinating.

Sam is right - - there are many Hesperian Health Guides that can offer much useful information.

While they come as large PDFs, the program Sam recommended, www.online2pdf.com/pdf-reduce-size, can considerably reduce their size. PDFs of 4 MB size will take a quarter of a day to transmit at 20 MB/day, but will be useful. You folks at Outernet should consider them. Ken

This is exactly the type of content we want to send, but simply reducing the size of the PDF is not sufficient. We need these as text files, which compress far better.

Could you provide a how-to so people can upload them to uplink.outernet.is? Also do you expect the 100kb restriction to remain in place? Or is that just for testing? I expect you’d get more interesting files with a 1Mb limit…

Just as Sam did, I reduced the 9.7 MB Where There Is No Doctor PDF this week to a 3.7 MB compressed PDF, then to a 1.1 MB text file and ZIPPED into a 365kB file. The text file lacked readability due to its paragraphing structure and lack of illustrations. It would take a large effort to fix I think, but it could be rebuilt. Here’s a screen shot:

I also created a PDF for each page and got 503 small (less than 100 kB on 90% of the pages) PDFs, but it would be a nightmare to transmit and then use at the other end.

I did not try Adobe’s subscription export program because I don’t have it. Perhaps then Hesperian folks might want to share their basic pre-PDF content with you, or you might want to “bite the bullet” and devote part of a day transmitting compressed Hesperian PDFs (which for this title would be on the order of 3.7 MB). Ken

We will gladly send that pdf. Feel free to send me the current smallest version of the PDF and let’s see how long it takes and whether it gets received.

Wonderful - - I’m going to send you 3 PDFs (by e mail later today) for the first 3 Hesperian Guides. I am pulling them off my Rachael Pi from World Possible http://worldpossible.org/

Ken

I took the first 3 files from the Hesperian collection (shown at the bottom of my screen print) and reduced them with a program I found called smallpdf.com, then with the program Sam recommended OnLine2PDF.com.

The first three files (with .compressed in the title) were done with smallpdf.com, the next three files (with Lowest in the title) were done with OnLine2PDF.com using the lowest resolution possible, and the final three files are the originals.

I’m sending the compressed files to Syed to take a look at, and see if he wants to upload some of them. The OnLine2PDF.com produces hard to read illustrations, but the format and text are fine. The smallpdf.com produces reasonable results but with less compression.

Ken

World possible looks great. @Syed have you spoken to them? Looks like they could be a great partner?

Aside from anything else it looks like they’ve already done a lot of work on collating content: http://dev.worldpossible.org/cgi/rachelmods.pl & http://rachelfriends.org/previews/rachelpi_64EN_4.0/

Could do worse than just broadcasting bits of that to start with, alongside the time-sensitve weather and news where outernet excels.

I think in terms of TRAFFIC management sending a large .pdf will just cause a unnecessary Log-Jam.

If we want to send that type of content it needs to be broken up into smaller chunks.

Either say .pdf chapters and then put back together or multi part zip.

Multi-part Zip has the disadvantage that you can not look at the file until you have all the parts.

The third and most sensible is to implement multiplexing software at the transmitting end and and reverse multiplexing at the receiving end.

That way many files can be sent at the same time, based on a priority structure which will determine which goes first.

Many routers do this, i.e. voice packets first, video next, HTTP next and non time sensitive last i.e. ftp of downloading bulk files.

In the Outernet 20 MB a day case the the data Multiplexer hierarchy might be:

News Ticker = first /Real time

Major News Update = within 60 seconds

Tweets=fast / minutes

Text Emails= Soon-ish / within 5 min

Hourly news report files = within 60 minutes.

Popular Webpages requests= Within 4 hours

Email attachments = within 24 hours.

Background bulk files = Last / Whenever.