Development of a multimodal sensing network to capture feelings and assessment of quality of life

Author: Valentino Šafran

Mentor: izr. prof. dr. Matej Rojc univ. dipl. inž. el.
Co-mentor: dr. Izidor Mlakar univ. dipl. inž. tk.
Degree: 2.
Date: september, 2021
DKUM: VALENTINO ŠAFRAN

Abstract

In the master’s thesis we present the development of a multimodal sensing network for the purposes of the European project H2020 PERSIST, which includes data or feelings, in order to assess the quality of life of patients. Before developing the system, we reviewed the process of capturing data from existing systems, and reviewed selected building blocks for our system. We also present the standards and protocols that are used within the proposed system. For the architecture of a multimodal sensing network, we selected three basic building blocks that we estimated could form a powerful network for data transmission and processing, and offer the possibility of upgrading in case of increased project requirements. These three building blocks are Apache Camel, Apache ActiveMQ Artemis and Apache Kafka. We put all the widgets on the PERSIST_CAMEL physical server, each into its own virtual machine. Machine learning microservices, with the exception of questionnaires conducted by Rasa Chatbot, which are in the process of development, represent the clients of the system. The multimodal sensing network is adequately secured with secure protocols and the use of other security features. Finally, we present the results of testing the load and responsiveness of the system, when answering the questionnaires. The results show that the set system successfully implements data streaming for a given number of 200 patients in the PERSIST project, and can also support a larger number of users.

FIWARE platform

Machine learning and neural networks have been booming in recent years. Evaluating confidence in neural networks is an approach by which we obtain information from the deep neural network about the certainty of recognition or perception. With this additional information, we can improve the performance of the model by excluding from the final set of outputs those with a low degree of certainty. Cloud services offered by technology giants such as Google, Amazon, Facebook, Apple and the like are easily accessible and cover data that is in itself very useful to providers as it helps in the development of services. For users’ access to this data on such systems, however, the story is quite different. It is necessary to pay providers for access to data and services, so the system is closed to them in this respect. To address such problems, the FIWARE system has been developed at the initiative of the EU. FIWARE is a modular system built on open-source components in order to accelerate the development of smart solutions. Smart Solutions are a combination of innovative hardware and software technologies in the field of the Internet of Things. The core of the FIWARE system is the Orion Context Broker, which enables real-time data management and sharing. In this way, Smart Cities can share information about events on city streets, such as traffic information, air quality, free parking spaces and location information. Similar to parcel and mail delivery companies, they can share order information such as e.g. current location and expected delivery time. This information, which describes what is currently happening, is called Context Information. The FIWARE context broker is essentially the RESTful API (Application Programming Interface) and allows applications to provide new information to the system and gain access to certain system context information.

Building blocks of the system

Apache Camel is an open source framework based on the Java programming language and the well-known Enterprise Integration Patterns (EIP). Camel allows you to define routing and forwarding rules in a variety of Domain Specific Languages (DSLs), including a Java-based Application Programming Interface (API) and Spring or Blueprint XML configuration files. This means that in the Integrated Development Environment (IDE), either in the Java editor or XML (Extensible Markup Language) language, routing rules or routes can be defined in a “smart” way. Camel focuses on simplification and supports easy integration of systems for developers. Spring is the framework and container for the Java platform. The basic functions of the framework can be used by any application in Java. Spring Boot makes it easy to create stand-alone Spring applications, even without prior programming knowledge, which can also be run easily and quickly. Most Spring Boot applications require a minimal Spring configuration in simple text format. Spring also offers fast and reliable resolution of security issues. Spring developers are very active in working with security experts to correct reported vulnerabilities, given that this is an open source framework. Dependencies from other vendors are also closely monitored, and regular updates are issued to make data and applications as secure as possible. In addition, Spring Security simplifies integration with standard security schemes and provides reliable solutions that are secure by default. REST (REpresentational State Transfer) is based on web standards and uses the Hypertext Transfer Protocol (HTTP). It is based on resources, where each component is presented as a resource. The resource is accessed via an interface (API) using standard HTTP methods. REST was first introduced by Roy Fielding in 2000. The online service is a collection of open protocols and standards used to exchange data between applications or systems. Software applications written in different programming languages and running on different platforms can use online services to exchange data over computer networks, such as the Internet, in a manner similar to interprocess communication on a computer. This interoperability (eg between Java and Python or Windows and Linux applications) is due to the use of open standards. Online services based on the REST architecture are known as RESTful online services.

Swagger, or nowadays called OpenAPI, is an Interface Description Language (IDL) and is used to describe RESTful APIs, by using the JSON format. Swagger is used in conjunction with a set of open source software tools to design, build, document and use of RESTful web services. Swagger includes automated documentation, code generation, and test case generation. HL7 FHIR (Health Level Seven Fast Healthcare Interoperable Resources) is a standard and specification in the field of health, written in the Java programming language. FHIR belongs to the domain of digital electronic records (Electronic Health Records – EHR), where data is created as resources. FHIR allows you to create, edit, delete, and share medical resource definitions for specific profiles, such as Patient, Observation, Questionnaire, and more than 140 other resources. Each resource can be presented in XML, JSON, ND-JSON and RDF format. Apache ActiveMQ Artemis is an open source project to build a powerful asynchronous group messaging system based on many protocols. The use of asynchronous communication makes it easy to connect clients and intermediaries in a given small-sized messaging environment. MQTT is a very lightweight protocol for sending messages from client to server, based on the publish-subscribe principle and supports asynchronous communication as described in the MQTT specification. MQTT was designed specifically to reduce the size of network traffic and code footprint, or resource consumption on client devices. This makes the MQTT ideal for the rather limited resources of small devices with sensors and actuators. Apache Kafka is a distributed event streaming platform that can handle trillions of events daily. Originally designed as a message queue, Kafka is based on a commit-log. The logbook records past system events, which monitors system operation and makes it easier to troubleshoot errors within the system. Since it was created by LinkedIn in 2011, Kafka has rapidly changed from a message queue to an event streaming platform.

Multimodal sensing network

As part of the European PERSIST project, a multimodal sensing network architecture was proposed. The architecture is connected by two VLANs, which are located on two servers. On the first server, called PERSIST_CAMEL, there are several virtual machines that host widgets, such as: Apache Camel, Apache ActiveMQ Artemis, Apache Kafka, clients for testing all widgets, and micro service questionnaires. Another server, called PERSIST_INFERENCE, houses machine-based microservices such as: automatic speech recognition, text-to-speech, and recognition of patients’ facial emotions from video diaries.

Testing

All the building blocks of a multimodal sensing network are located in their virtual machines, on two physical servers. Both servers have a 1 Gb Internet connection within VLAN A. A server named PERSIST_CAMEL is also connected to an internal VLAN B with an internally accessible subnet and hosts five virtual machines via the Proxmox VE 6.3-2 virtual environment. Xubuntu 20.04 LTS is installed on this server. There are no virtual machines on the other server named PERSIST_INFERENCE. On this server, on which Ubuntu Server 20.04 LTS OS is installed, microservices are planned: FER, ASR, ECA and TTS. The infrastructure specification presented in Table 5.1 can be said to be lightweight with 8 GB of RAM for Apache Camel version 3.4.0. 32 GB of RAM is allocated to Rasa Chatbot version 2.1, which requires more memory. Each widget has one socket with four processor cores and a 32 GB SSD.

Discussion and results

Arhitektura večmodalnega zaznavnega omrežja (VZO) za projekt PERSIST.

Development of a multimodal sensing network to capture feelings and assessment of quality of life

Discussion and results

Laboratory of Digital Signal Processing

E-mail

Contact

Address

Phone