System Monitoring

bmcintosh · 2015-07-19T22:44:22.331Z

Now that I have played a bit with the system, I am getting a handle on how it all works. I have another thread going regarding my first receiver.

Once the system gets some users that are really using the data, it is important that the system is ‘up and running’ all the time.

This means we need to know if the data stream heading into the unlink is running OK, and that receivers in the footprint of those satellites that carry the data are receiving OK.

If we had a couple receivers that could monitor the presence of data on the receive end, and have those receivers themselves monitored for proper operation (i.e. have them send regular data via some port from the monitoring station), we could then test the stations once in a while with something like Zabbix, to ensure the Pi is actually up and running.

Then, we could simply report that the Pi is receiving data - IPC calls?.

The ‘monitoring station’ would need to have a back end internet connection that is pretty stable, and some battery backup on the satellite, pi and networking so that a local power failure would not take down the monitoring station.

Basically, I am talking about a simple network monitoring setup. This will become more important when people start to use the outernet data for real, but it is something to look at.

Cheers,
Brian

branko · 2015-07-20T08:07:24.694Z

We already have the monitoring code and the server component that emits warnings via email. We will be deploying them soon and provide an updated image that starts up a monitoring daemon.

tjanos · 2015-07-20T10:36:58.234Z

I think, a global monitoring of outernet receivers can based on a twitter-bot:
On the twitter need an account for example #outernetreceivermonitor
All to sended tweets can forward and broadcast on outernet sat channels.
On the ORx receivers need a running cron job, collects informations about the receiver status, and form a twitter message, and if it is online, send it to the twitter in regular time base. Maybe it is a good starting point to listen a local monitoring daemon.
disadvantege:
it is working only if the receivers are online.

branko · 2015-07-20T11:13:51.041Z

This is the current implementation: GitHub - Othernet-Project/monitoring: Signal motoring for Outernet service

Comments and suggestions are welcome.

tjanos · 2015-07-20T20:37:17.028Z

Branko,
thank you for the info about your monitoring package.
It seems, it is a confortable, working solution for remote monitoring one/some outernet receivers.
I took a short loook-around in the code, and it seems me, to make a working setup not too easy for me.
The client part is a python program, monitor.py, it is working on the RPi new system without any additional setup or requirements.
But to use it as a real data source, it need some additonal paremeters at the start. Here is its help:

ORxPi v0.2.4 | root@orxpi:/home/outernet/monitoring-master/client> python monitor.py -h
usage: report ONDD internal state to remote server [-h] [–url URL]
[–key PATH]
[–socket PATH]
[–buffer PATH]
[–platform NAME]

optional arguments:
-h, --help show this help message and exit
–url URL, -u URL remote server URL
–key PATH, -k PATH client ID key file
–socket PATH, -s PATH
path to ONDD socket
–buffer PATH, -b PATH
path to data buffer
–platform NAME, -p NAME
platform name

It seems, at least it need an url to the remote server.

Without this parameters it put the output regularly in a file, here: /tmp/monitor.buffer
One of an example contents of it is here:

ORxPi v0.2.4 | root@orxpi:/tmp> cat monitor.buffer
[{
“processing_time”: 0.062773942947387695,
“timestamp”: 1437420656.8720491,
“sat_config”: “d6267d3”,
“pid”: “65”,
“snr”: 0.0,
“client_id”: “6ce4bfb8-6875-426f-a0fb-82fc1d144b2d”,
“transfers”: true,
“bitrate”: 87640,
“signal_lock”: true,
“platform”: null,
“service_id”: “outernet-0”
}]

===========
It seems me, this is the output of some carefully selected IPC calls to the ONDD daemon.
To experiment with is, it would be enough time-to-time to send this file to another machine to evaluate it manually.
But I think, you have a working server to accept this records, it would be good to learn more about this. If you allow to send/redirect some of this record to such a server, it would be very usefull for us.
Maybe you have a lot of another jobs and tasks, and this monitoring system is not your first priority.
So, thanks again to the code and the info.
t.janos

branko · 2015-07-21T05:57:09.441Z

The source code repository also includes the server, which you can run on your local machine and let the receiver send data to it via LAN. We intend to put up a server at a public location by the end of this week.