Issues receiving Weather.html


#1

I have been using Outernet for a little while now. So far I have received many things including community data, news, Wikipedia articles (some of them pretty large), APRS feed and the weather grib2 files. However, the Weather.html (downloading as d4e8-Weather.tbz2) has never gotten past 98% or 99%. I have tried deleting everything in /mnt/cache but it eventually pops the partial file back in there with about 600kb already filled in but says it is starting at 0% on the status page.

My SNR varies between the high 3s and the high 5s with an average around 4.

Here is some additional diagnostic information:

Storage usage:
/boot                  25%      23.6M
/                      37%      38.1M
/mnt/conf               0%      29.4M
/mnt/cache              0%     532.3M
/mnt/data               4%     895.7M
/mnt/internal           1%       1.4G
/mnt/downloads         1%
/mnt/external           1%       1.4G

Uptime:
 22:53:33 up 3 days, 11:26,  load average: 2.16, 2.42, 1.98

platform:

version: 3.1

build: 81529a1

build time: 2016-10-30 18:20:01+00:00

And a screen shot showing my current status…

Let me know what else could help troubleshoot this.


Very good results on my RPi 3 setup...and a question
#2

It will eventually get there. Just be patient.


#3

I’ve seen others say the same thing. I’d just like to understand why when I delete the entire cache folder it downloads many things (seems like they were stuck) then starts out by creating a ~600KB file and shows the weather file at 0%. It slowly downloads until it gets in the 90s for percent complete then stops. The first time I left it alone for almost 4 days. Only today did I start trying to delete the cache and see if that helps. Each time I delete the cache a bunch of new data downloads until it gets stuck on Weather again.

Is there an explanation on how the carousel works so I can understand what is going on? Or maybe a debug mode of some sort? Other large files download fine, see below.

This is while downloading…

And completed…

The Wikipedia article grew from just a few bytes to the size seen in the cache folder in the first picture above. This is how all other files seem to behave.

Oh and here is the Wikipedia file in its proper place:

-rw-rw-r-- 1 postgres postgres 495972 Jan 3 00:10 /mnt/downloads/Wikipedia/Fergie_(singer).html


#4

The other large files are “newer” and thus have higher priority than the Weather.tbz2 file. So they go around the carousel more often that the Weather file does. Also, higher priority files can interrupt lower priority files (so the news updates every hour for example will temporarily interrupt Weather). Weather is pretty much the only file that is “permanently” on the carousel.

Its still surprising you have not received the Weather.tbz2 after three days. I am going to re-up its priority. It should not take more than about a day.

That said, it looks like you are fairly technically proficient. You should try the Skylark release - it has the Weather.tbz2 built in and thus does not need to download it off the air.


#5

Well that makes more sense. Is the carousel proprietary or is there some sort of document or diagram explaining how it works down to the packet level? Just curious.

Yes I am fairly technical and would like to help with this project as time permits. I know python well enough to fix small things but I am better at C/C++/C#. My background also includes networking technologies down to the frame level (and layer 1 of course).

I figured the Skylark release had it built-in, but part of the enjoyment of this has been receiving data from a satellite. :slight_smile: Anyway since you adjusted the priority I’ll wait a day just to see what happens. If it is still not making good progress then I’ll drop Skylark on there. Any specific feedback you guys are looking for with Skylark?


#6

:thumbsup:

Re skylark: We are mainly looking at improving the stability and performance compared to Librarian. As you might have noticed by now, Librarian has pretty bad performance. It also degrades quite badly as the number of files downloaded increases.

So general usability, bugs, stability, performance.

One advantage Librarian still has over Skylark is, give thats its a basic UI, its very usable on Mobile, while Skylark currently is not. That is still a work in progress.


#7

Sounds good. I haven’t noticed the performance hit yet.

I’ll upgrade to Skylark later this week, just want to give it one last chance to pull down that file. :slight_smile:

I will miss browsing on my phone. It may seem silly but I enjoy reading the news and other items instead of just accessing them on the Internet directly.


#8

kf4hzu. Read the exchange on this thread and it helped me understand what was going on with the weather download. It was my understanding that Skylark won’t be available for the RPi 3 - am I correct? If it is planned also for the RPi, I’ll replace Librarian based on this discussion.

Thanks .

Richard
KE7KRF


#9

I’ve been experiencing this same problem. d4e8-Weather.tbz2 had been trying to download for almost a week. But I have been patiently waiting for Skylarks full release.


#10

Ditto. I have been stuck at 99% since Saturday 12/31. Getting plenty of other data, so hopefully re-uping its priority will resolve this shortly. Looking forward to useful weather information.


#11

No luck here…

Time for Skylark! :slight_smile:


#12

On a side note, my community file showed up! :slight_smile:

Jan 4 06:56:01 rxos user.info ondd[13271]: [carousel] completed: opaks/b523-two-transistor-radio.gif.tbz2


#13

Adding my name to the list, I’ve been running continously since about December 27, the weather download very slowly built up to 99% and seems stuck there for the last few days. At first I would SSH in and reboot if it looked like it was stuck (which seemed to be the case at 66%), I stopped doing that after reading the forums, and it eventually got up to 99%, but seems to be unable to get past that point, even though other files seem to come down just fine. It’s been stuck at 99% for about 3-4 days, and as you can see below it’s run for 5 days without reboot.

Of note, even though most of the time I get a strong signal, anywhere from 5-8 db SNR, due trees to the south (and possibly weather), it occasionally drops below 3db and will drop a few packets. You can see in the current run I’ve received 380320 packets and dropped 242 over the last 5 days. Various other files seem to interrupt the Weather download and AFAICT they complete just fine (including quite a few grib2 files, news files, wikipedia articles etc, see ‘du’ list below).

I’m running rxOS 3.1 on a RasPi 3, battery backed against power failure. Using the outernet patch antenna, nooelec SNA and RTL-SDR.com v3 SDR.

Some diagnostic info
[rxOS][[email protected]:/mnt/cache]$ uname -a Linux rxos 4.1.21-RXOS #6 SMP PREEMPT Sun Oct 30 23:53:51 IST 2016 armv7l GNU/Linux [rxOS][[email protected]:/mnt/cache]$ uptime 02:15:13 up 5 days, 0 min, load average: 0.45, 0.47, 0.66 [rxOS][[email protected]:/mnt/cache]$ df -h Filesystem Size Used Available Use% Mounted on devtmpfs 457.7M 0 457.7M 0% /dev /dev/mmcblk0p1 196.9M 189.5M 7.4M 96% /boot overlay 100.0M 13.8M 86.2M 14% / tmpfs 462.7M 28.0K 462.7M 0% /dev/shm tmpfs 462.7M 172.0K 462.5M 0% /run /dev/mmcblk0p5 27.4M 417.0K 24.7M 2% /mnt/conf /dev/mmcblk0p6 551.0M 5.9M 504.8M 1% /mnt/cache /dev/mmcblk0p7 1.8G 160.5M 1.6G 9% /mnt/data /dev/mmcblk0p8 25.7G 127.5M 24.3G 1% /mnt/internal /mnt/external;/mnt/internal 25.7G 127.5M 24.3G 1% /mnt/downloads /dev/mmcblk0p8 25.7G 127.5M 24.3G 1% /mnt/external [rxOS][[email protected]:/mnt/cache]$ ls -lat |head -20 total 5592 drwxr-xr-x 3 root root 12288 Jan 5 01:47 . -rw-r--r-- 1 root root 162474 Jan 5 01:47 zindex -rw-r--r-- 1 root root 809442 Jan 5 01:47 b7bddf9d7382a961010ec78dbe5a58d28848ca58f6f0589ff6e5583b8b65e529 -rw-r--r-- 1 root root 1675 Jan 4 21:41 19beedfa132c43f593022e7510473aa58fa20e1895fa1a0e9cb949c0be79b362 -rw-r--r-- 1 root root 969 Jan 4 18:46 dde0c21280f33c55a295f54ed60d306542f05ce514cccf9227283c8a04405f8a -rw-r--r-- 1 root root 2815 Jan 4 18:32 9fc5d0308fc412d9580ec95b7938d2989e02a4f6fe3668d71845561dc97736e5 -rw-r--r-- 1 root root 2205 Jan 4 17:53 175edfd2fca9070843547ef91b26147b0050b0013e1883fa490d0e30a3fe2fc1 -rw-r--r-- 1 root root 1663 Jan 4 17:41 d508e85e9ddad8a4ce6fc910bd20f1f7137557b4ebabef28e28b9416f0357184 -rw-r--r-- 1 root root 2874 Jan 4 16:47 c894df6a1e74388b656941a6854411ea613909ed87af451a744c2ca9dabf2fd6 -rw-r--r-- 1 root root 2685 Jan 4 16:46 c526a3b33eef8233fc1e11517a92797575038729f035e084b1592c4835e5b3b9 -rw-r--r-- 1 root root 3340 Jan 4 16:46 fce4e3381699c298c0b4f42a7fd552b91e56954ee99f7c6e86f3606b9d3254c9 -rw-r--r-- 1 root root 2801 Jan 4 16:45 3f83c48800e8a6d34a5ad24b10af4be253b7065e106b841c8ef5dadb5a8c7d6b -rw-r--r-- 1 root root 3706 Jan 4 16:43 83b71e4a20137d5b87c1b5db4ea9e75e87be678fbfbddaf0a61231837406aeca -rw-r--r-- 1 root root 1973 Jan 4 16:42 e2342fbd6399621f54b244cf8d5b727e2be7e50e26d986081c46dd3767c07dfe -rw-r--r-- 1 root root 2326 Jan 4 16:10 0e293e53ccbf582b583133db31bdbb1fda6dc01c3c2e0ce28649826770980ff4 -rw-r--r-- 1 root root 2653 Jan 4 15:56 882c5698c655f0b85658396d0223aa684b714393e2e5b930163607aec47fa8ca -rw-r--r-- 1 root root 2508 Jan 4 15:55 027ffe03da7a704d2f47ebf8fdd1066093cd9e46e21fb3c705936b24306ca72a -rw-r--r-- 1 root root 2754 Jan 4 15:54 5e3cfd4919fc95557bc1f50f35782091e3a26ee5285d6f50fdd6630919e6d3aa -rw-r--r-- 1 root root 2470 Jan 4 15:53 0b0ef275cd6a63037064e3e1945cbcb41567392d03de21e0125625ed97dab1b4 [rxOS][[email protected]:/mnt/cache]$ cat /proc/meminfo MemTotal: 947592 kB MemFree: 228464 kB MemAvailable: 624268 kB Buffers: 109400 kB Cached: 326624 kB SwapCached: 0 kB Active: 498208 kB Inactive: 174536 kB Active(anon): 269608 kB Inactive(anon): 9920 kB Active(file): 228600 kB Inactive(file): 164616 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 52 kB Writeback: 0 kB AnonPages: 236708 kB Mapped: 42320 kB Shmem: 42820 kB Slab: 29964 kB SReclaimable: 17168 kB SUnreclaim: 12796 kB KernelStack: 1112 kB PageTables: 2516 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 473796 kB Committed_AS: 565500 kB VmallocTotal: 1105920 kB VmallocUsed: 2132 kB VmallocChunk: 823324 kB CmaTotal: 8192 kB CmaFree: 3736 kB [rxOS][[email protected]:/mnt/cache]$ cd /mnt/downloads [rxOS][[email protected]:/mnt/downloads]$ du -h . 4.0K ./opaks 12.0K ./.sop 68.0K ./Community Content/.thumbs 280.0K ./Community Content 16.0K ./lost+found 10.2M ./Weather/data/oscar 2.1M ./Weather/data/weather/current 2.1M ./Weather/data/weather/2016/12/30 2.1M ./Weather/data/weather/2016/12/31 2.1M ./Weather/data/weather/2016/12/28 2.1M ./Weather/data/weather/2016/12/25 1.8M ./Weather/data/weather/2016/12/24 2.1M ./Weather/data/weather/2016/12/27 2.1M ./Weather/data/weather/2016/12/26 2.1M ./Weather/data/weather/2016/12/29 16.3M ./Weather/data/weather/2016/12 16.3M ./Weather/data/weather/2016 2.1M ./Weather/data/weather/2017/01/05 2.1M ./Weather/data/weather/2017/01/04 2.1M ./Weather/data/weather/2017/01/01 2.0M ./Weather/data/weather/2017/01/02 2.1M ./Weather/data/weather/2017/01/07 2.1M ./Weather/data/weather/2017/01/03 2.1M ./Weather/data/weather/2017/01/06 14.4M ./Weather/data/weather/2017/01 14.4M ./Weather/data/weather/2017 32.7M ./Weather/data/weather 42.9M ./Weather/data 692.0K ./Weather/grib2/gfs.2017010200 336.0K ./Weather/grib2/gfs.2016123100 688.0K ./Weather/grib2/gfs.2016122600 344.0K ./Weather/grib2/gfs.2016122700 688.0K ./Weather/grib2/gfs.2017010100 684.0K ./Weather/grib2/gfs.2016122900 1.3M ./Weather/grib2/gfs.2016122400 684.0K ./Weather/grib2/gfs.2016123000 684.0K ./Weather/grib2/gfs.2016122800 340.0K ./Weather/grib2/gfs.2017010400 1.3M ./Weather/grib2/gfs.2017010300 7.6M ./Weather/grib2 50.5M ./Weather 20.8M ./Wikipedia 8.0K ./preprocess/sop 12.0K ./preprocess 12.0K ./Amateur Radio/AMSAT/ARISS 12.0K ./Amateur Radio/AMSAT/News 28.0K ./Amateur Radio/AMSAT 2.8M ./Amateur Radio/APRS/APRSAT 2.9M ./Amateur Radio/APRS 2.9M ./Amateur Radio 4.0K ./grib2 8.5M ./News 83.6M .


#14

no, you are right. there is definitely a problem with the Weather.tbz2.

I am still working on it, though at this point its clear that the experiment with transferring an important component of the system - the weather app - over the satellite link, has not been very satisfactory.

Thats why the weather app is preintegrated in the new Skylark firmware.

To checkout the Weather app in Librarian, you can get a copy of the Weather.tbz2 from:

https://archive.outernet.is/images/Weather/

Place the tbz2 file in the “opaks” directory, give the system a few minutes to take care of it, and then you should be able to use the weather app.

Of course, at this point I am encouraging everyone to switch to the Skylark firmware.


#15

Thanks, I’ll give that a try in a few hours and report back


#16

Abhishek,

(Sorry, I just got back to this)… How many minutes do you think? I copied the file out there a couple hours ago. It showed up immediately in the web interface for the opaks directory, but it doesn’t look like its been picked up. Reboot needed?
[rxOS][[email protected]:/mnt/external/opaks]$ date Fri Jan 6 21:56:46 UTC 2017 [rxOS][[email protected]:/mnt/external/opaks]$ ls -lat total 832 drwxr-xr-x 2 root root 4096 Jan 6 21:47 . -rw------- 1 root root 682220 Jan 6 19:45 Weather.tbz2 -rw-r--r-- 1 root root 159744 Jan 6 18:36 26f5-oscar_vel8854-surface-currents-oscar-0.33.json.tbz2 drwxrwxr-x 12 postgres postgres 4096 Jan 4 16:01 ..


#17

In my personal instance it took a reboot. Of course I could have messed something up that contributed to it not loading right away


#18

Ok, problem solved, but it was non-obvious (and still not 100% sure why). Posting here in case someone else is stuck in the same predicament.

Rebooting seemed to do nothing, fiddling with permissions and "touch"ing the file did nothing. After looking around at the /etc/incron.d configs, I took a guess that maybe how I put the in opaks was causing it not to trigger the opakhandler.

What I originally did (that didn’t apparently work): I scp’ed the file to the home directory, then cp’d (copied) it to “/mnt/external/opaks”.

The “fix” that caused it to get picked up was to instead mv (move) the file from home to the “/mnt/downloads/opaks” directory. Even though those appear to be mounted to the same place, either /mnt/external/opaks didn’t trigger the /etc/incron.d/opaks.incron task, or copying it (instead of moving it) didn’t.

Once I removed it and then moved (mv) it to “/mnt/downloads/opaks”, it decompressed the weather app and deleted the file in just a few seconds.


#19

So this happened…

Jan 7 04:56:15 skylark user.info ondd[417]: [carousel] completed: opaks/d4e8-Weather.tbz2


#20

kf4hzu

The weather app finally saw fit to completely download for me as well, about two nights ago. Really glad to have it and looks great on my Win 7 laptop - not so good on the iPod Touch.

Richard
KE7KRF