Application of the raspberry pi for In Situ measurement automation and data transfer and storage

Texto integral

Introduction

Monitoring of the marine environment is imperative to ensure the ecological safety of coastal zones, to control their resources and to study their current state and trends. Monitoring tasks entail continuous measurements and the obtained measurement data should be available at any time for further processing and analysis. To organise this observation process, it is necessary to create automated remote workplaces providing reliable connection to the measuring equipment for control and data reading without the participation of an operator. This is of particular importance given that it facilitates the modernisation of existing hardware and software complexes (HSC) for measurement, data recording and storage (e.g. portable weather stations, wave-graphs, etc.), which assume long-term autonomous operation, but lack their own built-in remote control, monitoring and configuration capabilities.

Methods of sea surface monitoring are discussed in detail in the literature [1–4], but the issues of organising remote workplaces for data recording, processing and subsequent storage are given much less attention [5]. Work [6] provides an overview of the development of system solutions for weather stations since the mid-1990s. It shows that the development of means for measuring environmental parameters coincided with the trend of development of telecommunication and computer technologies. These technologies of meteorological measurements are based on software and hardware complexes that register data from sensors and transmit them to a remote terminal via radio channel. The paper [6] shows variants of realisations of software and hardware meteorological complexes including solutions ¹, ², ³, ⁴, but not enough attention has been paid to the issue of the creation of automated workplaces itself. In fact, this direction (automation of workplaces) is diffused in the hardware and software complex of weather stations.

A more detailed and scrupulous approach to the issue of workplace automation is used in the development of software systems for the collection, visualisation, archiving, processing and transmission of hydrometeorological measurement data from automatic hydrometeorological complexes, in particular for sea surface monitoring. A notable example of such software is the ALMETA software package ⁵, a special software operating on a personal computer with Windows operating system (OS), providing collection and processing of hydrometeorological information.

However, its flexibility and functionality are excessive for specific tasks, such as in situ measurements of the marine environment. This is particularly problematic given its significant demand on the computing resources of a personal computer.

Technical solutions based on Arduino and Raspberry Pi hardware and software platforms require fewer resources. In [7–11], examples of Arduino-based weather station realisations are presented, and in [12–20] – Raspberry Pi-based. But these are hardware and software implementations of weather stations that record and process measurement results. The issues of automation and organisation of remote workplaces are hardly considered in these works.

In the Russian Federation, the Recommendations on the Operation of Automated Meteorological Complexes in Observation Units (R 52.04.818-2014) have been established, which set out the requirements for the installation, maintenance and operation of automated meteorological complexes (AMC), describe the standard procedure for meteorological observations in observation units (OU) equipped with AMC as well as the actions of OU personnel in the event of AMC failure. However, these recommendations do not permit to formulate technical requirements to the HSC, which are responsible for the automation of measurements and operation of the HSC in the autonomous mode. The purpose of the development is to create a remote automated workplace for continuous monitoring of the sea surface and environmental parameters using single-board computers. The paper describes a new technology of workplace implementation based on the hardware and software platform Raspbery Pi, used as a controlling personal computer (PC) of the weather station Davis Vantage Pro 2, and presents an example of its use for control and monitoring of the marine environment.

Equipment

The weather station Davis Vantage Pro 2 was chosen due to its measurement accuracy, functionality, number of settings [21] as well as presence of remote measurement sensors that provide radio transmission up to 300 m in the open space ⁶, which is relevant within the context of the MHI stationary oceanographic platform in Katsiveli. It is imperative to acknowledge that, in the absence of personnel on the platform at all times, the responsibility of remote access to control and retrieve data from the weather station Davis Vantage Pro 2 is of significant importance, as previously highlighted. This weather station allows connection to a PC using both in-house commercial WeatherLink software by Davis Instruments ⁷ and free WeeWX software ⁸.

The advantage of WeeWX software is that its source code is open (available for review, study and modification), which ensures its flexibility and adaptability

to specific tasks. This software works in various GNU/Linux systems, in particular Debian, Ubuntu, Mint, Raspbian and others. Raspbian support allows the Raspberry Pi to be used as a control PC for the weather station Davis Vantage Pro 2.

As mentioned earlier, the Raspberry Pi is most often used as a stand-alone solution for building weather stations, when the Raspberry Pi board itself is used as a central processor to which external sensors are connected [22]. These solutions are inferior to the weather station Davis Vantage Pro 2 both in measurement accuracy and in the flexibility of software settings.

It is rare for Raspberry Pi-based solutions to be used as a controlling HSC, with ordinary PCs most often being used for remote access and control. However, in conditions of limited access to the platform in Katsiveli and a lack of permanent staff, not only the performance of PCs, but also their reliability and energy efficiency to ensure stable autonomous operation, become critical. In this context, Raspberry Pi has been shown to exhibit superiority over ordinary PCs, with its power consumption typically not exceeding 15 W.

The following is an example of realisation of remote access and control of the weather station Davis Vantage Pro 2 using Raspberry Pi HSC as well as the results of its pilot operation on the platform in Katsiveli.

Results and discussion

At the initial stage of the remote access organisation, a Raspberry Pi B+ was used as a control PC to collect and transmit data from the weather station Davis Vantage Pro 2.

The choice of this model is due to its computing resources, the performance of which ensures stable operation of WeeWX software when receiving, processing and transmitting data to a remote server. The 700 MHz processor core clock and 512 MB of RAM of this HSC provide support for any version of Raspbian OS, in which the WeeWX software runs. Connection to the Internet is provided through the built-in Ethernet port. Wireless connection to the Internet can be made via a USB-modem – in our case TP-Link TL-WN727N. Four USB ports make it possible to connect the weather station Davis Vantage Pro 2 as well as a mouse and a keyboard (Fig. 1).

As previously stated, the WeeWX software incorporates built-in support for the weather station Davis Vantage Pro 2, facilitating the aggregation of data on a local PC, specifically a Raspberry Pi B+.

The data are presented graphically in Fig. 2 and stored in the data file weewx.sdb. It is possible to present measurement results in a similar manner on a remote server. However, additional settings are required for monitoring via the Internet.

One of the limitations of the WeeWX software when running in Raspbian is the use of FTP (File Transfer Protocol) for data transfer. Thus, it is important to choose a server that supports this protocol. FreeHostingEU (available at: https://www.freehostingeu.com) was chosen as such a server (hosting). On this server, an account with ID: 4089596 was registered, for which a third-level domain was created – vantagepro2.eu5.net, providing data display.

Fig. 1. The appearance of the workplace for automated measurements, transmission and storage of the weather station Davis Vantage Pro 2 data using a Raspberry Pi B+

Fig. 2. Example of view of data on pressure (a) and inside temperature (b) on the platform in Katsiveli from 16 November 2022 to 16 December 2022

The FTP server settings are made in the WeeWX software configuration file, weewx.conf, and are as follows:

[[FTP]]

#FTP’ing the results to a webserver is treated as just another report,

# albeit one with an unusual report generator!

skin = Ftp

#If you wish to use FTP, set @enable@ to @true@, then

# fill out the next four lines.

# Use quotes around passwords to guard against parsing errors.

enable = true

user = 4089596

passwort -= ************

server = vantagepro2.eu5.net

path = / vantagepro2.eu5.net/

#Set to True for an FTP over TLS (FTPS) connection. Not all servers

# support this.

secure_ftp = False

# To upload files from something other than what HTML_ROOT is set

# to above, specify a different HTML_ROOT here.

# HTML_ROOT = /var/www/html/weewx

# Most FTP serverts use port 21

port = 21

# Set to 1 to use passive mode, zero for active mode

Passive = 1

It should be noted that when organising automatic data backup using Raspberry Pi there were some difficulties, which were not considered by the technical support of the FreeHostingEU FTP server and were solved experimentally – by selecting the settings of the HSC. The path for saving data from the weather station Davis Vantage Pro 2 in the configuration file weewx.conf differs from the default path recommended on the FreeHostingEU technical support site – /home/www. In the settings of the weewx.conf configuration file, the path is specified as /vantagepro2.eu5.net/. If you leave it as it is written by default, the data will not be sent to the FTP server and will not be displayed on the page www.vantagepro2.eu5.net.

As a result of the above settings, data from the built-in sensors of the weather station Davis Vantage Pro 2 were obtained (Fig. 2). Measurement results of the Davis Vantage Pro 2 weather station during operation are continuously recorded by the WeeWX software into the weewx.sdb data file, which is stored on the local disc of the Raspberry Pi in the /var/lib/weewx/weewx.sdb folder. To process and analyse the results of measurements as well as their backup, it is advisable to copy this file from Raspberry Pi to another PC or network data storage. Fig. 3 shows the block diagram of the algorithm for connecting and saving data in the network storage.

Fig. 3. Block diagram of the algorithm for backing up a file with weewx.sdb data on Yandex Disk

In view of the above, it was decided to copy the weewx.sdb data file on a scheduled basis to Yandex Disk, a cloud service of the Yandex company that allows storing data on servers in the cloud and transferring them to other users on the Internet. Access to this service is possible through WebDAV (Web Distributed Authoring and Versioning), a set of extensions and additions to the HTTP (HyperText Transfer Protocol) which supports collaborative work of users on file editing and file management on remote web servers. For this purpose, davfs2 software was installed on Raspberry Pi, which makes it possible for Raspbian OS to connect to WebDAV storages (in our case – Yandex Disk) as if they were local discs. A folder (mount point) /mnt/yandex.disk/ has been created on the local disc of Raspberry Pi where weewx.sdb data file is copied with specified periodicity. This folder is

further synchronised with the folders of Yandex Disk network storage and thus the data are transferred to the cloud.

To work with Yandex Disk, a new account was created – vantagepro2@yandex.ru, with its parameters (login and password) specified in the corresponding file of the davfs2 software – /etc/davfs2/secrets. It should be noted that Yandex Disk makes it possible to set a password different from the account password (in our case vantagepro2@yandex.ru). Raspberry Pi configuration work has shown that it is advisable to do this. In this case, connection to the NAS (authorisation on it) is more stable and access failures are excluded.

Synchronisation of the contents of the Raspberry Pi local folder and Yandex Disk is provided using the built-in configuration file of Raspbian OS – fstab. This file contains information about different file systems and storage devices used by Raspberry Pi. It also specifies how the created mount point /mnt/yandex.disk/ will be used.

It is necessary to note an important point that arose when configuring the remote workplace. For Raspberry Pi, when configuring the fstab file for the /mnt/yandex.disk/ folder, it is important to specify the _netdev parameter. In this case, connection of Raspbian OS to Yandex Disk is performed only after connection to Wi-Fi or Ethernet network. If this parameter is not specified, Raspbian OS will try to connect to the remote network storage before connecting to the Internet. The attempt will be unsuccessful and it will be necessary to try to connect again manually after Wi-Fi or Ethernet connection is established.

The above settings are made once and stored in the Raspberry Pi settings.

Data copying – weewx.sdb file – was performed cyclically, twice a day in order to avoid data loss. The built-in Cron task scheduler utilised in Raspbian ⁹ was employed for this purpose. For automatic copying of data, an executable file (script) ya_sdb_update.sh was prepared, the contents of which are given below:

#!/usr/bin/env bash //A service line indicating the start of the script

sudo rm /mnt/yandex.disk/olweex.sdb //Deleting a file with outdated information

sudo mv /mnt/yandex.disk/weewx.sdb /mnt/yandex.disk/oldweex.sdb //Copying with renaming the file with data for the previous period

sudo cp /var/lib/weewx/weewx.sdb /mnt/yandex.disk/ //Copying the file with up-to-date data

This file implements the data backup algorithm (Fig. 3).

During initial setup, two data files were saved on Yandex Disk:

• sdb – for the previous period;
• sdb – for the current period.

In the course of executing the ya_sdb_update.sh executable file, the file oldweewx.sdb stored on Yandex Disk is identified as outdated resulting in its deletion. The weewx.sdb file contains the data preceding the measurement data, which is the data available at the moment of command file launch. In order to facilitate the restoration of the data, the weewx.sdb file is renamed to oldweewx.sdb. The actual data itself is written to the file weewx.sdb.

As mentioned earlier, data transfer was carried out in Wi-Fi network via TP-Link TL-WN727N USB modem. Connection to open Wi-Fi networks is performed through the graphical interface by selecting the network and entering the password. Connection to hidden networks requires additional settings in Raspbian configuration file – wpa_supplicant.conf. For each newly connected network the following entry is made in it:

Network = {ssid = “Network name” scan_ssid = 1 psk = “Password” key_mgmt = WPA-PSK //Type of encryption }

In the process of establishing a connection to the concealed Wi-Fi network on the oceanographic platform in Katsiveli, it is imperative to consider the scan_ssid parameter. This parameter is crucial as it instructs the system to establish a connection to the wireless network even in the absence of a name transmission from the network. In the absence of this parameter, the connection process is unsuccessful, and consequently, data transmission is not facilitated. It is imperative to note that the scan_ssid value has been set to one.

The experimental operation of Raspberry Pi B+ exposed a significant disadvantage inherent to this HSC: the absence of software and hardware mechanisms for remote control and configuration of the HSC.

The configuration of the wireless network in Katsiveli does not assume the use of static IP addresses for PCs connected to it. Therefore, it is difficult to use the system of remote access to the Raspberry Pi desktop using the RFB (Remote FrameBuffer) protocol. Whenever the HSC is restarted (e.g. in case of a power failure), its IP address will change randomly. To regain access to it, the new IP address must be learnt, which is difficult and, in some cases, can only be done by connecting locally to the HSC directly on the oceanographic platform itself.

In such cases, the optimal solution is the utilisation of specialised software for the remote control and configuration of HSC, in particular AnyDesk. The merits of this software are manifold, including its ease of installation and configuration, wide range of supported operating systems, autoloading functionality and ability to activate the HSC remotely over the local network (e.g. following a reboot or power failure) via the Wake-on-LAN function. However, it should be noted that this software is only compatible with Raspberry Pi 2 and subsequent models of this HSC. This was the reason for abandoning the Raspberry Pi B+ and switching to the Raspberry Pi 2.

The installation of AnyDesk on the Raspberry Pi 2B necessitated the implementation of additional preliminary settings. It should be noted that the AnyDesk

software package utilises the EGL graphical interface [23], which necessitated the configuration of the Raspberry Pi HSC. This entailed the installation of the additional software packages Libegl1-mesa and libminizip1, which were initially absent by default. It was only subsequent to the completion of these steps that the AnyDesk package on Raspberry Pi 2B commenced functioning correctly.

The WeeWX software and database backup settings for the Raspberry Pi 2B were configured in the same way as for the Raspberry Pi B+.

In the pilot operation on the oceanographic platform in Katsiveli together with the Davis Vantage Pro 2 weather station from August to December 2022, a PC in the following configuration was used:

• model – Raspberry Pi 2B;
• remote access software – AnyDesk;
• FTP server for data output – FreeHostingEU;
• server to store the data file – Yandex Disk;
• task scheduler (scheduled data file uploads) – Cron.

In December 2022, the MHI own FTP server – Asustor AS5304T – was put into operation. Additional backup of available data was organised on it. Taking into account the fact that there is no pre-installed HTTP-server and, accordingly, there is no possibility to display data in the form shown in Fig. 2, the contents of the data folder on Raspberry Pi B – /var/www/html/weewx/ – are copied to the specified folder of the FTP-server Asustor AS5304T. The Cron task scheduler is used to automate the process. Copying the contents of the /var/www/html/weewx/ folder was organised with the help of the cURL program pre-installed in Raspbian, which sequentially transfers all files from Raspberry Pi 2B to the /Home/www/html/weewx/ folder of the FTP storage. A new executable file (script) asustor_ftp.sh was prepared with the following contents:

#!/usr/bin/env bash //A service line indicating the script start

cd /var/www/html/weewx/ //Switching to the data folder

find -type f -exec curl -u username:password –ftp-create-dirs T {} ftp://ip_ftp-server:port/Home/www/weewx/ \; //Copying the data folder to an FTP server.

In order to facilitate the subsequent checking and processing of the measurement results obtained, it is necessary to copy the folder to a local PC and open it in an Internet browser in a form that is similar to that shown in Fig. 2.

In September 2023, a further workplace modernisation was undertaken. External sensors measuring temperature, humidity, wind direction and speed were connected to the weather station Davis Vantage Pro 2. The Wi-Fi wireless connection was replaced with a wired LAN connection in order to enhance the reliability of remote access. Disabling the Wi-Fi modem also led to a reduction in the power consumption of the Raspberry Pi 2B PC. This modification enabled the utilisation of an external power supply with a voltage of 5 V and a current of less than 2 A, which also had a favourable effect on the stability of operation. This configuration has been in operation at the workplace since October 2023. Fig. 4 presents an example of the measurement results obtained from the weather station Davis Vantage Pro 2 in January and February 2024.

Fig. 4. Data on outside temperature (a) and wind speed (b) on the platform in Katsiveli from 20 January 2024 to 19 February 2024

The graphs demonstrate the stability of a Raspberry Pi 2B PC-based remote workplace in terms of its capability to record and store the Davis Vantage Pro 2 weather station data.

Conclusion

An example of implementing a remote automated workplace to access and control the weather station Davis Vantage Pro 2 using Raspberry Pi HSC is described. This workplace allows continuous automatic data storage locally, directly on the Raspberry Pi 2B, as well as backup to external FTP servers and network storage.

The ability to access, control and modify the parameters of the Raspberry Pi 2B HSC remotely eliminates the need for constant technical specialists to be present on the oceanographic platform in Katsiveli.

The results of the Raspberry Pi 2B HSC operation on the oceanographic platform in Katsiveli have shown that the possibility of creating automatically executable files (scripts) inside Raspbian OS and providing stable remote access to this HSC increase the reliability of recording, storage and backup of data received by the weather station Davis Vantage Pro 2.

¹ Wilhelm, R. and Haupt, F.S., 2011. Patent no. USRE42057E1, Int. Cl. G01W 1/00 (2006.01), G01P 13/00 (2006.01). N 11/485,648. Weather station. Filed 13 July 2006; Date of Reissued Patent 25 January 2011. 7 p.

² Strebkov, D.S., Dorzhiev, S.S. and Bazarova, E.G., 2013. Patent 2472186C2 Russian Federation. МПК G01W 1/00 (2006.01). [Network of Autonomous Environmental Monitoring Stations]: 2011110187/28. Applied 18.03.11, published 10.01.13. Bulletin no. 1. 7 p. (in Russian).

³ Fisher, G.A., 2009. Patent no. WO2009015370A1, Int. Pat. Cl. G0IW 1/10 (2006.01), G04B 47/06 (2006.01), G0IW 1/04 (2006.01) N PCT/US2008/071259. Pocket Weather Station. Filed 25 July 2008, published 29 January 2009. 25 p.

⁴ Runge, T.H., 2016. Patent no. US9301460B2, Int.Cl. G05D 11/00 (2006.01), AO1G 25/16 (2006.01) Irrigation Controller with Weather Station: N 13/406,410. Filed 27 February 2012, published 5 April 2016. 25 p.

⁵ ALMETA, 2018. [Software Complex for Automated Hydrometeorological Complexes ALMETA. User Manual]. 47 p.

⁶ Davis Instruments, 2021. User Manual for Vantage Pro2 ™ and Vantage Pro2 Plus ™ Weather Stations. 58 p.

⁷ Davis Instruments, 1999. Davis Weatherlink Software User's Manual. 78 p.

⁸ Keffer, N. User's Guide to WeeWX. 2024. [online] Available at: https://weewx.com/docs/4.10/usersguide.htm [Accessed: 10 December 2024].

⁹ Hentzen, W., 2004. Cron Explained. Hentzenwerke Publishing Inc., 12 р.

Sobre autores

Andrey Rozvadovskiy

Autor responsável pela correspondência
Email: rozvadovsky@yandex.ru

Junior Research Associate, Ph. D. (Engin.)

Bibliografia

Arquivos suplementares

Arquivos suplementares

Ação

1. JATS XML

Baixar

2. Fig. 1. The appearance of the workplace for automated measurements, transmission and storage of the weather station Davis Vantage Pro 2 data using a Raspberry Pi B+

Baixar (17KB)

Metadados

3. Fig. 2. Example of view of data on pressure (a) and inside temperature (b) on the platform in Katsiveli from 16 November 2022 to 16 December 2022

Baixar (26KB)

Metadados

4. Fig. 3. Block diagram of the algorithm for backing up a file with weewx.sdb data on Yandex Disk

Baixar (57KB)

Metadados

5. Fig. 4. Data on outside temperature (a) and wind speed (b) on the platform in Katsiveli from 20 January 2024 to 19 February 2024

Baixar (36KB)

Metadados

6. Fig. 3. Block diagram of the algorithm for backing up a file with weewx.sdb data on Yandex Disk

Baixar (52KB)

Metadados