Citizen science to approach machine learning to society: Detecting loneliness in older adults

Abstract

Background

Even if we are not aware of it, machine learning techniques are part of our daily lives. It is of the utmost interest that citizens become familiar with the use of these techniques and discover their potential to solve everyday problems.

Objective and Methods

In this article, we describe the methodology and results of a highly replicable citizen science project that allows citizens to get closer to the scientific process and understand the potential of machine learning to solve a social problem of interest to them. For this purpose, we have chosen a problem of social relevance in contemporary societies, namely the detection of loneliness in older adults. Citizens are challenged to apply machine learning techniques to identify levels of loneliness from natural language.

Results

The results of this project suggest that citizens are willing to engage in science when the challenges posed are of social interest to them. A total of 1517 citizens actively engaged in the project. A database containing 1112 texts about loneliness expressions was collected. An accuracy of 83.12% using the logistic regression algorithm and 62.23% accuracy when using the Naïve Bayes algorithm was reached in detecting loneliness from texts.

Conclusions

Detecting loneliness using machine learning techniques is an attractive and relevant topic that allows citizens to be involved in science and introduces them to machine learning practices. The methodology of this project can be replicated in other places around the world.

Introduction

Citizen science

Citizen science, also known as participatory research, gives all members of society the opportunity to take an active part in the research process. This type of science provides chances for enhanced public involvement and the democratization of science. In citizen science projects both professional scientists and citizen scientists benefit from participation. Citizen participation usually has an implicit or explicit educational value. ¹ Some citizen science projects offer citizens to participate in specific research stages such as data acquisition or data classification, while others allow citizens to participate at all stages of the scientific research. The crucial stage of a citizen project lies in the early phase of engagement, wherein volunteers must comprehend the project aims and ascertain their contribution. ² With regard to the level of, and efficacy, citizen engagement, the latter’s impact varies across fields. For instance, astronomy has long benefited from the work of amateurs. ³ Amateur astronomers make regular observations of the sky, identifying astronomical events such as supernovae, eclipses, comets, and asteroids. They have also made discoveries such as the recent discovery of comet Shoemaker-Levy 9. These observations often complement those made by professional telescopes. They are also actively involved in the popularization of astronomy, as is the case with the Amateur Astronomers Association. This in turn has led to the creation of tools that have brought science even closer to citizens. ⁴ Another research field that has recently benefited from citizens’ participation is earth observation science, ⁵ and ecology ⁶ in part due to the popularization of open hardware and software. ⁷ An example of citizen science in the field of ecology is the air watchers project. In this project, citizens participate by growing a strawberry plant in the windows of their homes. The strawberry plants function as detectors of heavy metals in the air. Once the strawberry plants have been grown, they are analyzed to determine the amount of heavy metals they have fixed and thus establish a map of the air quality in an area.

In contrast, psychology has not benefited as much from citizen sciences as other research fields. ⁸

Machine learning

Machine learning (ML) is a sub-field of artificial intelligence (AI). The objective of ML is to create methods that “learn” from data, and later they can be used to make predictions on new data. As presented in Deisenroth et al., ⁹ a model is said to learn from data if its performance on a given task improves after the data is taken into account. Common prediction tasks are classification and regression.

The objective of a classification task is to identify new data as belonging to a particular group among a predefined set of groups. An example of a classification task is a set of movie reviews that are classified as positive, neutral, or negative according to their valence; the classification task is to guess the valence of a never-before-seen movie review. ¹⁰ In this case, the data that the ML method learns from is a set of previously classified movie reviews, the data the ML method uses to classify the new review is its text, and the output is the group the review belongs to (positive, neutral, or negative).

On the other hand, the objective of regression is to estimate the unknown value of a variable based on the known values of other variables. For example, a regression task is given a set of houses with data about their sizes, number of rooms, age, and price, to estimate the price of a new house based on these predefined parameters. ¹¹ In this case, the data the ML method learns from is the information about a set of houses with the data for each one (size, number of rooms, age, and price), the data the ML method uses to estimate the price is the size, number of rooms, and age of the new house, and the output is the estimated price of the new house.

Loneliness as a social issue

Loneliness is a multidimensional and subjective psychological construct. Perlman and Peplau ¹² defined it as “the unpleasant experience that occurs when a person’s network of social relations is deficient in some important way, either quantitatively or qualitatively.” Surkalim et al. ¹³ carried out a meta-analysis on the prevalence of loneliness around the world based on studies from the year 2000 up to 2019. They observe that loneliness at a problematic level is a common experience around all countries. ¹³ In later life feelings of loneliness are a common experience. A recent meta-analysis estimates that around 25% of older adults (>60 years) living in high-income countries experience some degree of loneliness at least some of the time and 1 in every 12 older adults experiences severe loneliness. ¹⁴ Furthermore, with the COVID-19 pandemic, the prevalence of loneliness in the general population, and of older adults in particular, has worsened.^15–17 For decades, the literature has noted that having high or long-lasting levels of loneliness correlates with physical and mental health problems. Leigh-Hunt et al. ¹⁸ conducted an overview of 40 systematic reviews addressing loneliness’s impact on health. They observed a significant correlation between levels of loneliness with increased all-cause mortality, poor mental health, worse general health, lower well-being, unhealthy behaviors (e.g. use of tobacco), and higher rates of suicide and dementia.

Loneliness has become a matter of public health interest in the whole world. Governments from around the world, such as the UK and Japan, have already initiated strategies to tackle this health-related issue that negatively impacts modern societies and their citizens. ¹⁹

In light of these data, there is an urge to raise awareness of the prevalence of loneliness among health care professionals, policymakers, and the general public, ¹³ as well as to implement strategies that improve its detection on the general population and older adults in particular.

Citizen science in the ML field

One of the first online citizen science projects is iNaturalist, ¹ which started in 2008. The iNaturalist project is a web-based data crowd-sourcing of observations. Users of the platform upload observations of organisms at any particular place and time. ²⁰ After an organism is properly identified, it becomes part of a large database of labeled organisms, which in turn, is used to train ML algorithms for automatic classification. ²¹ Citizen scientists contributing to iNaturalist provide data, benefit from trained ML algorithms to identify organism species, and can use public datasets sourced from the community in their own projects.

Another pioneering project is Zooniverse, ² which started in 2009 as the Galaxy Zoo project. Zooniverse started as a web platform aimed at classifying the morphology of galaxies with the help of volunteers. ²² The growing popularity of the Zooniverse platform made its founders decide to host projects from other fields such as ecology, biology, and the humanities. Volunteers contributing to Zooniverse projects are asked to label images as belonging to some predefined classes. In this way, data is provided by researchers, and volunteers perform human-based computation tasks. ²³

Franzen et al. ²⁴ highlight the role of citizens when participating in problems considered computationally intractable. For instance, when creating data sets with correctly tagged data to feed algorithms. One example where it has been applied is in counting the wildlife that is present in aerial survey images. ²⁵ This is called “collective intelligence” to reflect that some intractable problems can be addressed collectively by human beings. In our case, our hypothesis is that loneliness detection can have a computational treatment by creating a labeled dataset through the participation of citizens. Moreover, offering citizens a tool with which to experiment with this dataset will allow them to approach the field of ML through a problem of high social interest such as the detection of unwanted loneliness.

Citizen science in the psychology field

In the field of psychology, where the main interest of scientific research is the human mind and behavior, most empirical research involves by default the participation of the population in the process of data collection, but usually only at this stage and in a passive way. In this field, co-creation and co-production approaches are popular and commonly used. These are distinct but related approaches to citizen science. Co-creation involves researchers working with stakeholders, such as patients or community members to design and conduct studies that are relevant and responsive to real-world needs. Co-production involves stakeholders in the research process, decision-making, and implementation, promoting empowerment and ownership. Citizen science engages the general public as active participants in scientific endeavors, using the collective intelligence and resources of diverse communities to address research questions and generate data. Citizen science emphasizes the democratization of research, enabling broad participation and knowledge generation. The public involved in this type of research is broad. Psychologists can increase the relevance, rigor, and impact of their work by involving citizens as active participants in research. Recent psychology research employes citizen science in their investigations. A study about the population’s emotional responses to COVID-19 where citizens are involved in proposing the measures and research questions, disseminating the project’s aims and results, and collecting data. ²⁶ In the research on adolescents’ risk behaviors during COVID-19, where young citizens actively participated in the operationalization of the measured behaviors, the questionnaire development, or in the writing of the recruitment texts. ²⁷ Another example is an investigation regarding environmental epidemiology, in particular about how air pollution impacts people’s mental health. In this study, citizens were involved in all the phases such as sharing concerns, raising qualitative insights, or formulating the research questions. ²⁸

Project’s motivation

Although AI, and ML in particular, is widely used by most citizens on a daily basis (e.g. spam filters in email, recommendation systems, and car navigation systems), many of them see these technologies as black boxes, and something difficult to understand and to work with. So, the first motivation for this project is to bring citizens to the realm of ML using an engaging issue. We believe this project will contribute to the AI literacy of citizens. ²⁹

The second motivation of this project is to find a well-known problem that could benefit from the use of ML and that is attractive to citizens. We chose the social problem of loneliness in older adults because it is a current and relevant problem in today’s societies, which has been exacerbated and made more visible by the COVID-19 pandemic.

Objective

The aim of this citizen science project is to make ML accessible and appealing to citizens by applying it to a real-world problem. The detection of loneliness in older adults is a well-known health challenge in today’s societies. This challenge could benefit from ML techniques. As so, this project focuses on involving citizens in the application of ML techniques to the health problem of loneliness among older adults.

The solution proposed in this project is a chatbot that using natural language processing (NLP) recognizes the speech patterns that indicate whether or not a person is expressing loneliness in a natural conversation. This real-life application attracts citizens to get involved in the citizen science project. All citizens are invited to participate in the project and learn about how ML techniques can help to improve he health issue of detecting loneliness among older adults.

Methods: The Serena project

Ethical matters

Every phase of the project was executed in accordance with the Declaration of Helsinki ethical principles and was approved by the Jaume I University’s Ethics Committee (reference: CD/58/2019). No informed consent was requested from citizens who participated in any phase of the project by the institutional review board, as no data that could identify participants was collected during the project. Apart from the national and European Union regulatory ethical frameworks observed by the project team, the latter also adhered to certain broader ethical principles. For instance, attention was given to the robustness and safety of the ML model, to ensure that supervision was put in place to mitigate the risk of errors during the screening process that could lead to harm (e.g. misdiagnoses of loneliness). Additionally, in alignment with the spirit of the citizen science approach, the project actively engaged participants in a meaningful co-creation process. This meaningful participation aimed to ensure that the project met the needs of participants and respected the diverse perspectives they brought into the project. It is important to note that the data provided by citizen scientists is the same data that they will use for creating an ML project. The only computational resources needed to perform all tasks are personal computers, no big computational infrastructures are needed. In this way, our project follows the recommendations given by Ceccaroni et al. ³⁰ about developing citizen science projects in the era of AI.

Project implementation

The Serena project is addressed to promote citizen science participation in research using ML. As so, its phases cover a typical workflow of ML: (i) data acquisition, (ii) data cleaning, (iii) ML algorithms selection, (iv) algorithms training, and (v) algorithms testing.

The project was developed in Castellón City and was open for online participation. It lasted over one year, starting in September 2019 and ending in December 2020. The COVID-19 pandemic began during the first phase of the project, which required adaptations to facilitate online citizen engagement.

The Serena project presented in this paper has five phases: (i) data acquisition, (ii) data cleaning, (iii) crowd labeling, (iv) experimenting with ML, and (v) impact and diffusion of the citizen science project. The data acquisition phase engaged citizens and allowed them to participate in the project by contributing real texts that could be used to train the chatbot. The data cleansing stage was implemented to procure a dataset of anonymized quality texts. During the crowd-labeling phase, citizens were involved in classifying texts into two categories: those that convey loneliness and those that do not. The fourth stage allowed individuals to interact with the ML algorithms and test them using citizens’ previously collected real texts or their own texts. The final phase involved diffusing the project to engage citizens in it; this phase was active throughout the project’s duration.

This project aims to make the scientific process accessible to the general public. Almost all phases are open to anyone interested in getting involved. The only limitation was in the data collection phase, which was restricted to individuals over the age of 55 with access to a smartphone, as they are the target population for the technology developed in the project. This was done to ensure that the collected texts are representative of the sample for which the application is intended.

This section describes the methods of each phase of the project were citizens where involved.

Data acquisition

The first aim of this phase was to obtain two types of data: a dataset containing natural language texts of expressions of loneliness, and a dataset with punctuations from a loneliness scale that reflect the degree of loneliness experienced by the person expressing the text at that particular moment. These data will be used in future phases to train an ML algorithm to detect loneliness from a natural text. The second aim of this phase was to explain to citizens the process of data acquisition and the characteristics of a quality dataset required to train an ML algorithm.

A data collection tool was developed for this phase to collect people’s genuine expressions of loneliness and their scores on a validated loneliness scale. These two types of data are associated, indicating that each collected text corresponds to a score for loneliness. The tool includes the digitized version of the University of California Loneliness Scale (UCLA), ³¹ which is a validated scale used to measure loneliness in multiple languages. ³² Additionally, it includes other complementary questions described below. The detailed questions are described in the Supplemental Material.

• Demographic data: Age, gender, and cohabitation status.

• Questions from the 3-item UCLA loneliness scale. (e.g. How often do you feel that you lack companionship?)

• An open-ended question that allows collecting natural expressions of user’s feelings. (Finnaly, please let me know how your week has been.)

The tool is accessible through Google Assistant as a chatbot and was developed by researchers using Dialog Flow, a Google tool. The dialog with the chatbot consists of the described questions. This tool allowed the citizens to provide texts by expressing their genuine feelings in relation to loneliness via voice or written text. Figure 1 shows the interface of this tool.

Figure 1.

Simulation of the dialog with the chatbot using Google Assistant to gather data regarding loneliness.

Google Assistant allows users to answer questions by speaking, typing on their mobile phone, or clicking on an item in a choice selection widget. Voice allows users to communicate with a mobile phone in a natural way, while interacting with the mobile screen for typing or clicking may preserve privacy in some situations. In addition, by means of this multi-modal tool, older adults with some kind of disability will be able to interact with the tool. If a person has difficulty reading the written text, the interaction with the tool can be using the voice. In the same way, if a person has difficulty listening, the interaction with the tool can be using the keyboard on the mobile phone.

The methodology of this phase is mainly based on organizing workshops with citizens. The workshops lasted for two hours and took place in a room equipped with WiFi to assist citizens in using the tool. Additionally, a projector was available for presenting the project’s slides. These workshops had four objectives: (i) to show citizens how to interact with the chatbot, (ii) to collect data, (iii) to bring ML closer to the population, and (iv) to collect feedback on their experience with the chatbot. To this end, the workshops consisted of three parts. The first part was the presentation of the citizen science project, where they could learn about ML techniques and how these can help detect loneliness. The second part was practical, where citizens learned how to access the chatbot using their smartphones. They were then given time to talk to it and contribute to the dataset collection with their texts and the UCLA loneliness scores. By teaching them how to access the chatbot, they could continue to contribute with data whenever they wanted to and spread the project to other citizens. In the last part of the workshop, they were asked to give feedback on their experience with the chatbot. This is a way of involving them in the design and co-creation of the tool and improving its usability.

Only older adults aged 55+ with access to a smartphone and who were interested in participating in the project were included in the sample for this phase. This inclusion criterion was set to improve the validity of the collected data, mainly the natural expressions of loneliness. It was important to collect texts that were most representative of the final users of the tool. The sample size was not calculated as the aim was to obtain the highest possible participation to impact a greater number of citizens and to collect as many texts as possible to train the chatbot.

In the face of the pandemic situation, it was decided to provide citizens with an alternative tool to provide data for the chatbot training. This new tool is an online questionnaire developed using Qualtrics ³ platform. By means of this tool, questionnaires can be digitized and made public on the Internet through a link, which can be shared on social networks. Figure 2 shows a screenshot of such a tool. It is an alternative for citizens’ participation in the data acquisition phase, which may not be familiar with chatbots and Google Assistant.

Figure 2.

A screenshot of the data acquisition tool developed using Qualtrics. It shows the items of the UCLA test and the options selected by a person.

This tool was distributed via the project website, Facebook, Twitter, mailing campaigns, radio programs, and media coverage in newspapers and magazines. To increase citizen participation in this project phase, manuals on how to access the chatbot were published on the internet and distributed to older adults at organized workshops.

Cleaning process

The data cleansing stage aimed to remove all personal data from the text dataset. Additionally, irrelevant data was eliminated as it may negatively impact the accuracy and success rate of the model. Eliminating such data is critical to increasing the efficiency of the output and maintaining the privacy of citizens.

A team of five researchers with backgrounds in psychology and engineering was responsible for this stage. The team established three criteria for the cleansing process. Firstly, all identifiable personal information was removed in accordance with ethical standards. Secondly, any errors resulting from speech-to-text recognition were corrected. Third, all the texts without relevant content, such as single words or test phrases, were excluded.

The texts gathered in the previous phase were compiled into a spreadsheet. The five researchers conducted an independent and blinded assessment to determine whether a text should be included, modified, or excluded based on the three criteria. Subsequently, the researchers convened to discuss the decisions made for each text. In cases of disagreement, the researchers discussed each case until a consensus was reached. Cases of disagreement were considered only when multiple researchers had an opposing view from the majority. If four researchers deemed a text valid and only one disagreed, this was not classified as a disagreement, and the text was approved.

Crowd labeling

The first aim of this phase was to acquire a classified dataset that could be utilized to train the ML algorithms in detecting loneliness through NLP. The texts collected in the previous phases had to be divided into two categories to train the algorithms: texts expressing loneliness and texts not expressing loneliness. The second objective of this phase was to teach citizens about the crowd-labeling process when using ML techniques.

During the previous data acquisition phase, each text was stored alongside a loneliness score from the UCLA Loneliness Scale. This scale has a cut-off point that determines whether a person is feeling lonely or not at a certain moment. According to the UCLA score and its cut-off point each text could be classified into one of the two categories if it is assumed that a person who scores loneliness on the validated UCLA scale will express texts that convey the presence of loneliness. Therefore, this UCLA score serves as a basis for determining whether a text conveys loneliness or not.

Nevertheless, the UCLA score may not consistently indicate whether a text reflects loneliness, as an individual’s degree of loneliness may not align with the emotional undertone they convey. Moreover, various factors may have contributed to this, including potential issues with the phrasing of the open-ended question utilized to gather the text or the likelihood of some individuals not being forthright about their genuine emotions (e.g. when someone was merely testing the tool out of interest). Therefore, the crowd-labeling phase includes an additional stage, beyond the association of UCLA scores with texts, where citizens actively participate in text classification. This stage serves two purposes: first, to train the chatbot, and second, to educate the general public about the application of ML to the social problem of loneliness.

A crowd-labeling tool was developed to engage citizens in the classification of texts. This tool enables to utilize the “collective intelligence,” also known as the “wisdom of crowds” ³³ of citizens to improve the accuracy of the ML training. The tool allows one to present a text from the database to as many individuals as possible and then requests them an assessment regarding the presence of loneliness in the text. As so, during the crowd-labeling phase, citizens classified the texts into two categories: those communicating loneliness and those that did not.

The texts that passed the cleaning process were made available to citizens via the Qualtrics crowd-labeling tool (Figure 3). This tool was created using the Qualtrics platform. Figure 3 shows a screenshot of the interface for labeling texts. It was created to allow citizens to read a text and decide whether it conveyed loneliness or not.

Figure 3.

A screenshot of the labeling tool. Using the Qualtrics tool, a sentence is presented to the citizen scientists for selecting “isolation” or “absence of isolation” in the sentence.

When a participant enters the Qualtrics platform, a text is displayed. This text is randomly chosen from the dataset containing all filtered texts (see Figure 3). Participants are then asked to choose between two options according to their criteria, depending on whether or not they think loneliness is present in the text. After each labeling step, participants can decide to label a new text or leave the tool. This allows participants to label as many texts as desired.

The aggregate quantity of responses from citizens in the crowd-labeling phase, where they classify a text into one of the two categories (loneliness or not loneliness) was taken as a convergence to the true value, which in this case is the UCLA loneliness score. ³⁴ The citizen criteria were intended to be used as an independent measure of validation for the UCLA score. However, because of the low participation of citizens at this stage of the project, mainly due to the difficulties associated with COVID-19, the UCLA scores were ultimately used to label the texts.

Experimenting with ML

Once the texts in the dataset were cleaned up and classified, the next step was to make them available for classification purposes using ML algorithms. The first objective of this phase was to train an algorithm to detect if a text is expressing loneliness or not by using NLP. The second aim of this phase was to provide citizens with an easy-to-use tool for experimenting with ML algorithms and to understand its functioning in the practical case of detecting loneliness from natural texts.

For this phase, a tool was developed. The main requirements for this tool were: (i) to be easy to use; (ii) to be highly available to citizens; and (iii) to be easily upgradeable. The Node-RED ⁴ platform meets all these requirements and, in addition, it can run on devices with limited computational and storage capabilities, such as Raspberry Pi and BeagleBone Black. Node-RED follows the Low-code/No-code programming paradigm. Tools following this paradigm allow users to create computer applications without writing any code using a computer language. Instead, a graphical tool is provided to create the applications. In particular, Node-RED follows the flow-based paradigm, in which the information flows through a set of connected nodes, where each node makes some transformation on the input information, and provides the new transformed information as output. This way some nodes can be connected, each one performing some task on the information that traverses it. The final result is transforming the input information into some output data.

Node-RED offers the possibility to develop and share libraries developed by the community through an open portal. Node-RED libraries provide nodes with new capabilities (see Figure 4). The developed package is hosted as a Node-RED flow. ⁵

Figure 4.

A screenshot of the machine learning (ML) tool. This tool is based on Node-RED. A package with nodes implementing a well-known ML algorithm was developed.

In addition, developed Node-RED flows can be easily shared with others through the Node-RED flows library. ⁶

A series of face-to-face workshops with citizens had been planned for the duration of the project. However, the outbreak of the pandemic caused by COVID-19 made it impossible to carry out the planned workshops. As an alternative to the face-to-face workshops, a series of videos were created to show how to use the Node-RED tool together with the implemented library presented in section “Serena’s technical architecture.” These videos show, in a clear and concise way, how to use the ML algorithms implemented on the data acquired during the development of the project. In addition, it is also shown how to use the ML algorithms on any other dataset that is of interest to citizen scientists.

Impact and diffusion of the citizen science project

The impact and diffusion phase has been active throughout the development of the project, aiming to strengthen scientific culture in society and actively involve citizens. The dissemination was carried out in two formats, online and face-to-face. However, due to COVID-19, some actions had to be adapted to the online format, and others had to be canceled as they could not be done online (e.g. participation in the European Researchers’ Night).

Next is a description of face-to-face dissemination activities and their impact. In order to disseminate the project, 13 meetings were held with health professionals (nurses), representatives of centers that offer services to the elderly (non-governmental organizations (NGOs), specialized elderly care centers, humanitarian organizations $\cdots$ .), and representatives of the public sector (City Hall and Councils, Provincial Council). As described in the section on data acquisition, three workshops were held with older people with the aim of disseminating the project and teaching them how to interact with the chatbot. These workshops had a pedagogical and didactic approach and non-technical language, intending to establish communication between researchers and citizens and allow discussion. Four workshops were organized with psychology students at the Universitat Jaume I. The aim was to validate if the chatbot was a reliable tool to administer a loneliness questionnaire, ³⁵ to disseminate the project and to train students in data collection using chatbots. Students were also asked to provide feedback and suggestions for improvement on the usability of the chatbot.

Serena’s technical architecture

This section shows the interrelation of the tools described in each phase of the project. Serena’s technical architecture is composed of three interconnected layers. The interconnection between these layers is shown in Figure 5:

Data acquisition layer. This layer provides all tools needed to acquire data coming both, from Google Assistant, or from the Qualtrics platform.

Service layer. This layer provides all tools needed for storing data and cleaning and labeling the data. It includes tools that provide implementation ofML tools and NLP tools.

Presentation layer. This layer is oriented to citizen science users, it provides a graphical user interface based on Node-RED for loading data and applying on them to different ML algorithms (see Figure 4).

Figure 5.

Technological architecture of the Serena project.

Results

This section presents the results achieved in each phase of the project. The results are related to the two main objectives: making ML accessible to citizens and training a chatbot to detect loneliness in natural language texts.

Data acquisition

Several workshops were organized in collaboration with the City Hall of two towns, various NGOs, and stakeholders from centers for the elderly. Restrictions in mobility due to the outbreak of COVID-19 had a direct impact on the development of the project. In Spain, meetings of people who did not live together in the same home were forbidden. As a result, planned workshops on how to use the Google Assistant tool had to be discontinued. Only three onsite workshops were possible. Two workshops took place at a local Senior Specialized Care Center for elderly individuals, while the third was held at the local Red Cross Organization Center. Additional training was provided to the computer teacher at the Senior Specialized Care Center, enabling the dissemination of project information to other groups of older adults. The results of citizens’ participation in the workshops are described in Table 1.

Table 1.

Numerical indicators of the participation of citizens in the phases of the project.

Citizen science phases	Quantity
Data collection:
Number of people who assessed the chatbot	644
Number of older adults attending the workshop on collecting data and using chatbots	35
Number of people who accessed the questionnaire in Qualtrics (chatbot replacement)	186
Crowd-labelling:
Number of people who have accessed the crowd-labeling tool	40
Experimenting with machine learning:
Number of downloads of the Node-RED package	442

A total of 1112 texts were obtained through Google Assistant and Qualtrics. This dataset contains natural texts expressing feelings of loneliness, each with an associated UCLA loneliness score. There were 442 texts with a UCLA score between 3 and 5, indicating the presence of loneliness. The total number of texts associated with a score between 6 and 9, indicating the absence of loneliness, was 670.

Citizens provided feedback on their experience talking with the chatbot. Some of the recurring suggestions from citizens were to improve the robot’s voice to make it more welcoming warm and human-like, and to address the various issues with Google Assistant’s speech-to-text voice recognition, for example, the speech-to-text processor interpreted that the person had finished and stopped the conversion when the person paused briefly. These suggestions have been considered to enhance the chatbot’s usability. Google Assistant’s automated voice was replaced with a friendly human recording. The changes to the functionality of Google Assistant were beyond the control of the researchers and could not be addressed.

Cleaning process

The results of this phase involved the removal of texts that did not meet the established criteria to ensure a high-quality dataset for training the chatbot. During the cleaning and consolidation process, 27% of the texts were subject to disagreement by one or more reviewers. The number of initial texts was 1112; 88 were removed because they contained transcription errors, 420 were removed because they did not contain relevant content, and 137 because two or more reviewers agreed that the text did not provide information. Thus, the final result of the cleaning phase was an anonymized database of 467 content-rich texts, which accounts for 42.00% of the total.

Crowd labeling

The results of citizens’ participation in the crowd-labeling phase are described in Table 1.

Only 40 citizens accessed the crowd-labeling tool to classify the texts. The number of classifications was too low to use them to label the texts. The degree of agreement between the citizens’ ratings and the loneliness scale score could not be determined, nor could the degree of agreement between citizens when ranking the same text. As stated in the methodology section, the scores from the loneliness scale were used to rank the texts into the two categories.

Experimenting with ML

Although it was initially thought that not conducting the face-to-face workshops would be a handicap for the dissemination of the use of the ML tool, the data collected in Table 1 shows that the number of downloads of the implemented library was very high. So it is finally considered that the information through the videos reached a large number of people.

Out of the 467 texts in the dataset, only those that all researchers agreed were used for classification. Eighty-four texts were deemed not applicable by one researcher, leaving a total of 383 texts. Of these texts, 70% were used to train the Naïve Bayes and logistic regression learning algorithms, 10% were used for validation, and 20% were used for testing, the results (see Figure 4) were that the Naïve Bayes algorithm correctly classified the 62.34% of the test texts and the logistic regression algorithm correctly classified the 83.12% of the test texts. These initial results show that it would be possible to use ML algorithms as a tool to assist professionals in screening for loneliness.

Next, we present an example of a text classified as “No loneliness” by the algorithm. On Monday morning my daughter came to see me and brought my granddaughter, we went to the park and she energized me for the rest of the week. (Original text in Spanish: El lunes por la mañana vino mi hija a verme y trajo a mi nieta nos fuimos al parque y metió energías para el resto de la semana.)

Next, we present an example of a text classified as “Loneliness” by the algorithm. My daughter hasn’t come to see me this last month because of the Coronavirus. (Original text in Spanish: Este último mes, con el Coronavirus, mi hija no ha venido a verme.)

Impact and diffusion of the citizen science project

The dissemination of the project was carried out in both online and face-to-face formats. The impact achieved by the diffusion actions in the project is described in Table 2.

Table 2.

Impact of dissemination activities on citizens during the project’s duration (14 months).

Impact and diffusion	Quantity
Number of students who took part in the workshops on the dissemination of the project and the use of chatbots	154
Number of stakeholders who attended the project presentation meetings	16
Number of visits to the project website	2682
Number of posts about the project on Facebook	25
Number of posts about the project on LinkedIn	12
Number of posts about the project on Twitter	9
Number of views of posts about the project on Facebook	2071
Number of views of shared posts about the project on LinkedIn	4930
Number of views of shared posts about the project on Twitter	2991

The online dissemination activities included the following. A project website ⁷ and a profile on the social networking site Facebook. ⁸ The project was disseminated through Twitter, LinkedIn, the Jaume I University web portal, and the radio and written press at regional and national levels. The project was registered on the European Portal for Citizen Science Projects ⁹ and included in the “Citizen Science Observatory Report”. ³⁶

Discussion

To the best of our knowledge, this is the first citizen science project that aims to approach ML to citizens. The use of ML in citizen science projects is increasing and it shows a promising potential. ³⁷ It has been incorporated into citizen science mainly in three ways: assisting or replacing humans in completing tasks, influencing human behavior, and improving insights. ³⁰ In these projects, ML is mostly used as a methodological tool to reach a greater objective. For example, using supervised learning and Convolutional Neural Network with the objective of improving the recognition of the shape and structure of galaxies. ²² The novelty of Serena’s citizen science project is that ML is at the core of the project and is the goal in itself, as the project’s objective is to make ML an interesting, friendly, and attractive topic for citizens to learn about.

This citizen science project may contribute to the AI literacy of citizens. ²⁹ A large number of citizens were part of the project and learned about ML and its applications. In total, 1517 citizen scientists, including older adults, the general population, students, and stakeholders, actively participated in the project’s phases. This is the main value and success of this work and we believe it can be attributed to three main reasons. First, we chose a well-known social problem in today’s society to attract citizens to the project. Loneliness is a matter of public health that affects countries from all around the world. The scale of the loneliness problem has alarmed experts, some of whom have described it as a social health crisis. ³⁸ Selecting loneliness’ detection as the problem to solve using ML techniques served to attract citizens’ participation in the project and consequently to spread ML knowledge to a wider community. Second, we used co-creation methods and participatory design with citizens. These techniques allowed the development of tools and procedures that were more accessible and inclusive for citizens. For example, it allowed adaptations in the chatbot tool to make it more friendly to older adults and people who suffer from a visual or hearing impairment. Third and finally, we believe that part of the success was due to the collaboration with stakeholders. The collaboration of town halls, associations, and other entities that work with social matters was crucial to increasing citizens’ awareness and participation in the project.

The tests performed with the learning algorithms, shown in the section “Experimenting with ML,” show an accuracy of 83.12% when using the logistic regression algorithm and 62.23% accuracy when using the Naïva Bayes algorithm. These percentages, although promising in principle, should be corrected by expanding the number of texts in the dataset. In addition, although the classification used was binary, the two algorithms used are probabilistic, which implies that a probability is obtained for each of the classes used. This probability could be used to establish a new “unsure” category if the probability of the most probable class does not reach a certain threshold defined by the user.

Our project helps to raise society’s awareness about the possibilities that ML can offer in the field of mental health. Specifically, it shows how ML could help health professionals identify people suffering from loneliness. AI has a lot to contribute to the mental health of an aging population, however, there is still a lack of awareness and even distrust and fear among a large part of society. ³⁹ We believe that citizen science projects such as this one are an excellent way to increase the ML knowledge base in society, make it more accessible to everyone, and help to demystify some of the dangers associated with it. Czaja et al. ⁴⁰ suggest that it is crucial that we think about how we should apply AI to an aging population. For example by identifying design features that are best suited to older people, or by incorporating datasets provided by heterogeneous people from all group ages. We believe that our project is in line with these recommendations and that it will contribute to more innovative and inclusive science.

We believe this project has a high degree of replicability. This statement is based on the fact that the tools used in this project are accessible in a large number of languages and that all of them can be used free of charge. Though many citizens are not familiar with ML techniques, these are present in many of our daily life activities (e.g. leisure recommendation systems or spam filters). We believe it is important to encourage initiatives that aim to make ML research and practice more accessible and friendly for everyone to learn about. Furthermore, this project uses the social problem of loneliness, which is a matter of public health that concerns the whole world. There is an increasing general awareness about the need to detect those people in situations of loneliness due to its negative impact on their health and well-being. As so, we believe that the detection of loneliness using ML techniques would be a topic of interest in most places around the world and therefore Serena would be a successful citizen science project that could be easily replicated.

In a recent review, ⁴¹ they note that the majority of scientific publications on chatbots applied to mental health issues have been conducted outside of medical research settings, with the risks that this entails. This citizen science project develops a technological solution to the health problem of loneliness. This citizen science project develops a technological solution that could help healthcare professionals screen people suffering from loneliness. This proposal has been developed and supported by a multidisciplinary research team of psychologists, engineers, and computer scientists. There is a need to create synergies between health and technology disciplines to develop high-quality solutions with real-life applicability.

The majority of citizens’ participation in the project was virtual, as the data was collected, transferred, and stored primarily using digital tools. In order to gain higher participation from citizens and a wider profile (e.g. people with low digital skills) we planned face-to-face activities. However, many of these had to be adapted to an online version due to the restrictions of COVID-19. The use of digital tools entails ethical challenges that should be taken into consideration. This project adhered to ethical principles and recommendations. However, a limitation of this work that we must comment on is the ethical challenges that may arise from using Google’s tools to collect data. Especially when the data are real expressions of feelings of loneliness, which are health-related private matters. In future work, we recommend the use of open-source tools where researchers are the sole owners of the data.

There are various studies that apply ML techniques to text analysis for the purpose of detecting loneliness. Badal et al.^42,43 investigated how NLP can serve as a valid tool for detecting documented feelings of loneliness. Yamada et al. ⁴⁴ have applied ML techniques to real-life audio recordings of people talking about everyday matters to investigate the relationship between acoustic, linguistic, and prosodic features with the UCLA loneliness scale scores. Other recent studies analyze the text from social media to detect loneliness, for example, using text mining from Twitter posts.^45,46 Other studies have addressed the detection of loneliness using ML with other types of data other than text. For example, using objective behavioral measures from individuals’ censored homes ⁴⁷ or looking into their mobility patterns. ⁴⁸

One limitation of our study was the restricted participation of citizens in the crowd-labeling and ML experimentation phases. Only 40 citizens participated in the classification of texts. We believe this was mainly due to the COVID-19 restrictions, which made it impossible to conduct face-to-face activities to teach and engage citizens. The low level of participation of citizens in the data tagging phase may be due to the limitations of COVID-19, however, another possibility is that citizens did not find the topic of loneliness detection interesting. It is important to note that in other citizen science projects, participants are typically more engaged when the topic is closely aligned with their interests, such as spotting animals or stars. As the involvement of citizens in the data tagging process was so limited, their assessment of the presence of loneliness in the texts could not be used to improve the classification of the dataset. However, it is possible that greater citizen participation could have improved the quality of the database by applying collective intelligence. As this is a fundamental phase of the research project, it is recommended that future studies employ alternative methods to engage citizens in the classification of texts. Potential avenues for engagement could include offering remuneration to citizens or disseminating information through older adult associations.

The researchers needed to address as many citizens as possible to obtain representative data on expressions of loneliness and a large number of texts to train the algorithms. However, it is important to mention that the sample size for this study was not calculated, which presents a weakness. Inadequate calculation of sample size can pose significant issues when studying a psychological phenomenon and when training an algorithm. In the study of loneliness from a psychological point of view, an inadequate sample size runs the risk of producing an unrepresentative sample, which compromises the generalizability of the findings. This undermines the credibility of the research and its ability to inform clinical practice and policy. Insufficient data can jeopardize the reliability and validity of study outcomes, hindering advancements in understanding and addressing psychological phenomena. From a technological perspective, the primary issue of not training an algorithm with sufficient data is the potential for producing a deficient, inaccurate, or biased model. Inadequate data can hinder the algorithm’s ability to capture the complexity of patterns within the data, leading to unreliable predictions or biased outcomes. This is because insufficiently trained models may struggle to generalize to new and unseen data, which undermines the model’s reliability and practical applications, such as loneliness diagnoses from text. Adequate data training is essential for developing robust and reliable ML models. Future studies should calculate the sample size necessary for addressing loneliness among older adults in a representative manner and gather enough data to develop an accurate algorithm.

When critiquing our study’s procedure, we acknowledge the data-cleaning phase. This phase aimed to refine the collected texts to eliminate potential errors and enhance the quality of input for training the loneliness-detecting algorithm. However, we recognize a potential drawback in this approach. Filtering out errors may unintentionally discard valuable psychological insights, especially considering the bidirectional relationship between loneliness and other characteristic symptoms of mental health issues such as depression or dementia.^49,50 These mental health conditions may influence language and expression, thus affecting text quality. The errors present in the text may hold diagnostic value for detecting loneliness. Thus, when removing them to streamline algorithm training, there is a risk of overlooking relevant and significant information that is crucial for identifying loneliness in older individuals through their spoken expressions. This highlights the importance of preserving such linguistic nuances in future data-processing approaches.

A limitation of this citizen science project is that the citizens were not involved in developing the initial research question nor in formulating the open-ended questions directed at collecting natural expressions of loneliness. It would have been valuable to involve them in this phase as well as to ask them in which phases of the project they were most interested in participating.

A future work idea is to translate the project into other languages to make it available to a greater population.

In the future, chatbots could potentially detect loneliness levels in older adults. This could provide round-the-clock support for those experiencing loneliness, offering greater accessibility and availability. Additionally, chatbots provide a confidential and private environment, fostering increased honesty in self-disclosure when expressing feelings of loneliness. The analysis of natural language patterns by this method would facilitate consistent detection of loneliness, without human biases. Furthermore, continuous monitoring would enable interventions to be made in a timely manner and integration with ML technologies would improve the accuracy of detection through techniques such as NLP and sentiment analysis. Thus, the use of chatbots could be a promising approach to addressing loneliness in older adults.

A future suggestion is to replicate the project’s methodology using another psychological construct that citizens may be interested in, such as anxiety or depression.

Conclusions

In this paper, we present a citizen science project aimed at bringing the field of ML closer to citizens. Although much of the project’s duration occurred during the COVID-19 pandemic, we obtained high participation from citizens. We believe that this high participation may be due, among other factors, to the choice of unwanted loneliness detection as a case study in the field of ML.

Citizens provided free texts (1112 texts), collected with the help of a Google Assistant-based chatbot and a Qualtrics-based online questionnaire, which were cleaned and validated to form a dataset with which to train ML algorithms (383 texts). Software tools have been developed, based on the Node-RED platform, allowing citizens to experiment with ML algorithms and the set of texts obtained with the help of citizens.