Supporting Remote Survey Data Analysis by Co-researchers with Learning Disabilities through Inclusive and Creative Practices and Data Science Approaches

Abstract

Through a process of robust co-design, we created a bespoke accessible survey platform to explore the role of co-researchers with learning disabilities (LDs) in research design and analysis. A team of co-researchers used this system to create an online survey to challenge public understanding of LDs [3]. Here, we describe and evaluate the process of remotely co-analyzing the survey data across 30 meetings in a research team consisting of academics and non-academics with diverse abilities amid new COVID-19 lockdown challenges. Based on survey data with >1,500 responses, we first co-analyzed demographics using graphs and art & design approaches. Next, co-researchers co-analyzed the output of machine learning-based structural topic modelling (STM) applied to open-ended text responses. We derived an efficient five-steps STM co-analysis process for creative, inclusive, and critical engagement of data by co-researchers. Co-researchers observed that by trying to understand and impact public opinion, their own perspectives also changed.

1. Introduction

We are a UK-based research group consisting of academic researchers embedded alongside members and staff from Heart n Soul, an organisation that focusses on “the power and talents of people with learning disabilities (LDs)”. All members of Heart n Soul identify as having LD¹ and/or autism but they are not required to specify a type of diagnosis or severity. As a diverse team comprising individuals with LDs and/or autism, together we performed a type of collaborative and participatory ‘co-research’ called ‘inclusive’ research [39] whereby people with LDs and/or autism have ownership of research and their concerns are at the forefront [4, 22, 39].

In ‘inclusive research’, similarly to participatory design methods [12], there are several ways to highlight the contributions of non-academic researchers with disabilities in the research process [40]. Here, we use the term ‘co-researcher’ to recognize the potential differences in perspectives and skills of academically trained researchers, and those with lived experiences of LD and/or autism but without formal research training [40]. Eight members of Heart n Soul were co-researchers. In line with a fully inclusive research process [34], co-researchers are also co-authors. Throughout co-research, the academic researchers were receptive and responsive to the needs, ideas, and concerns of Heart n Soul.

Researchers with LDs have the power to self-represent and self-advocate, and therefore methods and outcomes must be accessible [21]. In order to enable more meaningful participation [23] we worked with co-researchers to co-design an accessible online survey platform and launch a bespoke survey [3] and this work is detailed in a prior publication. In this paper, we describe how we collectively analysed and evaluated this survey data (‘co-analysis’) by incorporating inclusive, creative practices and other strategies to data analysis.

Crucially, co-researchers wanted to ask questions in ways that were important and meaningful to them, reflecting how they saw themselves, and in a style that met with their perspectives. As a result, the survey was designed as “a conversation with the public” [3]. Our survey platform supported co-researchers to ask questions as video/audio/photo and text. likewise, survey respondents could respond by uploading audio/video recordings, photos, or plain text. These features allowed for creativity, accessibility, and inclusion at the survey design, data collection and co-analysis stages. The survey also collected demographic information. It was launched in December 2019 and data collection is ongoing (with > 1500 responses).

Co-researchers decided on survey questions with the aim of reaching as wide an audience as possible. This in turn dictated the format of responses for analysis, leading to vast quantities of free text answers to open-ended questions. In turn, this generated data we considered suitable for machine learning (ML) approaches, and which might also be processed in ways that could be appropriate for co-researchers who have difficulties with reading and writing (e.g. data visualisation). Academics selected the most appropriate techniques to process responses. Primarily, we hoped that ML models would aid the diffusion of power differentials whereby the co-analysed responses were selected by a computer (and not academics), and evaluated together as a group.

Although people with LDs have been involved in research studies as co-researchers, we are not aware of many studies where they conduct data analysis. People with LDs are sometimes exposed to data obtained from people with LDs [36] with small sample sizes [31, 36, 37]. The data co-analysis stage described here took pace during the global COVID crisis. Due to the drastic changes this brought to the research process, in particular moving from a specially designed accessible research space to remote working in a digital environment², it was essential that co-researchers were able to join our remote meetings [19, 25]. In order to realize our primary aims of co-analysis, we chose two main approaches: a creative arts approach designed to assist co-researchers to understand and explore data, visualization and information representation through image making, followed by a machine-learning supported approach using topic visualization (specifically LDAvis [32]). This enabled us to explore and reflect on contemporary automated approaches, supporting large-scale data analysis in groups comprised of individuals with very different abilities and backgrounds. Most importantly, co-researchers decided on these approaches themselves through a process of exploration and reflection.

In this paper, we describe and evaluate on the co-analysis process whilst reflecting on how we utilised inclusive research elements and practices in achieving the study goals. We describe our experiences, discuss key challenges in work of this kind, particularly how perspectives of co-researchers with LDs drove the co-analysis process. Importantly, we consider themes of Change in Design and “Nowhere and Everywhere” in the context of our study. As such, it is an inspirational case study of a Change Through Design whereby co-researchers with LDs lead the co-research process with the aim to change public attitudes towards LD and critically explore current data science practices as a means to understand their potential impact in co-research.

2. Related Work

2.1. Attitudes towards people with LDs

The Royal Mencap Society (2015) reports some improvements towards community inclusion [30]. However, negative attitudes prevail amongst the general public, and people with LDs are often the target of hostility. Also, Mencap reports confusion as to what LDs constitutes, indicating widespread misconceptions of the capabilities of people with LDs. Attempts to tackle discrimination have targeted children, adults, specific groups that work closely with LDs (teachers, caretakers, doctors, nurses, social workers), and those with public influence such as the media. The results however cannot be extrapolated to the general population and the long-term impact of such small-scale measures is hard to grasp. Importantly, evidence shows that contact with LDs is key to changing attitudes and reducing prejudice [2, 14, 29, 30, 42].

2.2. People with LDs and the digital divide

Ofcom state [24] people with LDs are less likely to have a computer (69% vs. 85%) or smartphone (70% vs 81%), or internet access from home (80% vs. 89%), from work (15% vs. 31%) or from a publicly accessible computer (23% vs. 16%). People with LD are less likely to use the internet for communicating (76% vs. 92%), for information (59% vs. 73%) or to access public services (32% vs. 43%). Digital divides have been reported to be more visible in disaggregated disability groups [7]. Evidence suggests people with LDs primarily use the internet for communicating through social media platforms, “which allows them to feel ‘like everyone else’ and create a sense of belonging to a ‘normal’ community” [13]. Barriers can also be financial and economic, which may bear some correlation with societal attitudes and exclusion [26].

2.3. Inclusive research with people with LDs: data analysis practices

In terms of broad approaches to inclusive research with people with LDs, this project has combined elements of a collaborative group with some elements of a leading and controlling approach [1]. The literature review indicates most studies assume an inclusive research approach working with qualitative data (oral histories, interviews’ transcripts) not quantitative data (statistics, big data) [17, 27, 35, 39, 41]. They also mention the tensions between aims and needs of an inclusive/participatory project vs. the institutional demands and language of the academy [17, 21]. Data co-analysis is one effective way of allowing co-researchers to own the research process, addressing issues of matter to them, sharing some control over the process outcomes and collaborative input to results, making sure these are presented in a creative and accessible way representing their views and experiences [23, 40]. Also, data co-analysis may have potential when “the boundaries between data collection and data analysis are blurred, and the process is organic” [21].

2.4. Analysis of open-ended questions with automated approaches

Work exploring the potential of semi-automated coding approaches in qualitative research shows it may be useful for coding tasks where researchers already follow practices that are suitable for automation [16], with the caveat that researchers prefer to use automation once a subset of data has been examined manually. However, results from these studies indicate simple NLP approaches such as topic modelling can perform as well as human coders on inter-rater reliability measures [16]. Latent Dirichlet Allocation (LDA) is a widely used approach shown to be useful in automatic labelling of Topic models [11]. LDAViz [32] is commonly used in data science to inspect potential similarity and relevance in LDA models. However, there is no research combining these approaches in the context of data analysis by LD co-researchers, who are often not considered in visualization research [43].

3. Methodology

Following ethical approval, we ran a series of 30 recorded remote workshops using video conferencing software. We designed workshops to address specifically planned questions by engaging in activities with eight LD co-researchers. As sessions were centered around data analysis, workshops introduced data analysis methods to co-researchers in varied ways. These were continually evaluated by academics and members of Heart n Soul. A significant number of workshops focused on introducing and exploring data analysis. Others explored the analysis of specific survey data through creative arts approaches, and machine learning supported analysis with topic modelling.

A Team Ethnography [3] approach was used for observation through remote video recording, interviewing, and inductive data analysis. Team members coordinated, shared, and discussed observations/reflections. We obtained multiple accounts of events, adding to the validity of the qualitative research process [3]. The Project Manager recorded reflections on the Padlet platform, which the team used to record and annotate research data³.

Using creative methods, summary statistics and graphs derived in Stata 15⁴, we co-analysed and co-evaluated demographic characteristics of respondents. Then co-researchers selected an open-ended survey question for further co-evaluation of the responses. We utilised machine learning-based textual analysis for structural topic modelling (STM) [20] in R⁵ as the first step to our understanding of public attitudes towards LD. The summary of the entire co-research process, including the survey co-design, is presented in Figure 1

Figure 1. Summary of the co-research process including key stages.

3.1. The Survey

Our co-designed survey targeted the general public [3]. Demographic question⁶ appeared at the very end of the survey to maximize complete response⁷. The survey launched on 3rd December 2019, International Day of Disabled People. Together with partner organizations and peer disability organizations, we engaged with people via social media and professional networks. A week after release, the Guardian newspaper published an opinion piece by one of our LD co-researcher⁸. The article was read by > 11,000 people in week one, and was shared widely on social media (over 500 times directly from the Guardian website). It was endorsed by high profile figures. We relied on marketing, media and word of mouth to secure respondents for the survey.

3.2. Covid emergency data collection process

We had to include every co-researcher regardless of ability. The global COVID crisis posed significant barriers. We assessed the co-researchers experiences and access to digital technologies and the internet. Considering each co-researcher’s individual preferences and requirements, we provided co-researchers with appropriate equipment and internet access via iPads and 3G connections. We organised a series of 30 weekly meetings with co-researchers using Zoom⁹ and Jamboard¹⁰.

3.3. Remote Meetings

3.3.1. Using Zoom, Jamboard and blended approaches

Weekly meetings were held with co-researchers on Zoom. Despite reports of security issues and privacy problems in Zoom, it was the preferred choice of co-researchers following sessions designed to establish their preferences. Jamboard was used to present and record ideas. Due to diverse conditions of co-researchers, we adopted blended approaches, incorporating physical copies of research outputs. Packs were sent to co-researchers, which we make reference to in the results section. A meeting agenda was prepared prior to each meeting and co-researchers were telephoned individually to invite them. In addition, there was continual discussion with co-researchers reflecting on their involvement and hopes for the project.

3.3.2. Safeguarding

Meetings were preceded by a 15-minute open session (not recorded) to give co-researchers the opportunity to communicate ideas and concerns. Following this, to help induce a calm, relaxed atmosphere, we played a piece of Heart n Soul produced music known to the community as ‘Cheryl’s song’, with a recorded voice-over by one of the co-researchers. Throughout each meeting, a trained team member was on call in case a co-researcher wanted to discuss issues of concern. After each session, an independent team member reviewed recordings and offered feedback on safeguarding to inform follow-up meetings.

3.4. Methods for co-analysis of the demographic data

First, using creative methods, summary statistics and graphs produced in Stata 15, we co-analysed and co-evaluated demographic characteristics of respondents. Further graphs and other forms of visualization were produced in the same fashion through co-researcher engagement.

Co-analysis of demographics happened in four steps. In Step 1 (Exploration), demographics were shown to co-researchers using Zoom screen sharing. An academic researcher generated bar and pie charts using default Stata command¹¹ and schemes defining overall graph appearance. We discussed co-researchers’ preference for colors (e.g. white versus yellow background), bar appearance (e.g. vertical versus horizontal), label size and legend (e.g. using ‘answered’ instead of ‘responded’). In Step 2 (Improvement), we used the Stata grstyle command to customize graph appearance according to feedback received in Step 1. The improved graphs were presented to co-researchers using blended approaches. Co-researchers received an accessible letter (See Supplement No.1) and stationary. They were asked to re-design the graphs according to their understanding and interpretation. In Step 3, co-researchers continued co-analyzing demographics in Zoom. Those who created re-designed graphs presented them to the group. Co-researchers were free to engage with materials for any duration and we asked all to present their re-designed versions at every Zoom meeting. Step 4 involved the implementation of further changes in Stata12. Re-deigned graphs were presented to co-researchers in Zoom meetings and further physical packs (See SupplementNo.2). Re-designed graphs continued to be used as a prompt in follow-up discussions to remind ourselves “Who are the people who answered our questions?”. In this paper, we report on the full co-analysis process as applied to a selection of demographics which co-researchers engaged with the most: Ethnicity and Education.

3.5. Methods for co-analysis of the open-ended survey responses

Co-researchers selected an open-ended survey question for detailed analysis and evaluation. As answers could be supplied as media, a back-end was integrated into the existing platform where researchers could access, annotate and transcribe such responses, using JAM stack13 for performance and security. Transcription had to be as objective as possible. We utilised structural topic modelling (STM) [28] in R to process the transcribed responses. We selected STM over LDA because LDA performs poorly on relatively small datasets [28]. The read_dta command was used to read in the dataset. Data was processed using the textProcessor function including ‘stemming’ to obtain words’ language root, and ‘the bag of words’ method [20]. Words occurring in only one document were taken out of the analysis. Guided by a previous study with a comparable sample size, and an accessible description of the STM model [20], we ran a 10-topic model with no covariates for initial exploration of emerging topics and the associated words using the Spectral method of initialization which has been reported as more stable [28]. Using the toLDAvis function, we exported the results to the LDAvis package [32] with the visualisation of the topic-word distributions and relationships. We focused our co-analysis on the most common topic (Topic 1).

Co-analysis of the open-ended survey question selected by co-researchers happened in five steps. All STM outputs were added to Jamboard as images and presented to co-researchers for evaluation. Initially (Step 0), for a more meaningful and equal upcoming coanalysis, we asked an artist affiliated with Heart n Soul to creatively explore with co-researchers how the survey data gets processed within the STM model, the concepts of ‘stemming’ and ‘the bag of words’ in particular. In Step 1, first, co-researchers were asked what they hope that people would answer to generate a hypothesis. Second, they were presented with top three associated words for the most common topic (Topic 1) to initiate conversation. In Step 2, we visualized the outputs using a word cloud for a more detailed conversation about the results. For Steps 1 – 2, co-researchers received a ‘creative pack’ with the invitation to think about, interpret and re-design the presented results at home (See SupplementNo.3). Co-researchers were free to engage with the materials for any duration and we asked all to present their re-designed versions at every Zoom meeting. In Step 3, five (out of 10 derived) most representative responses of the most common topic were presented to co-researchers using the STM plotQuote function. Through arts-based approaches, an underlying theme for the topic was further explored which reflected co-researchers’ new ways of understanding the public, themselves, and their place in the world with ethical implications for research practices. In Step 4, we explored the most common topic utilizing LDAvis with the possibility to co-analyze the remaining topics. Finally, in Step 5, co-researchers evaluated the entire co-analysis as part of the co-research process: what they did and how they feel about it. Co-researchers’ creative re-designs are presented along the STM outputs in the results section.

4. Results

4.1. Staying connected: ensuring access of co-researchers to digital technologies (Table 1)

Table 1. Digital devices and Internet access – co-researchers’ (Co-R) journey towards remote communication in Zoom.

Co-R	Own internet (WiFi)	Devices currently14 used	Communication platforms used (project-relevant)	Our immediate actions	Results(including changes)
1
(David)	YES	Phone, iPad, laptop	email	Sent a new iPad 7 with 3G to ensure he has access to good connection for meetings.	Internet connection issues (at times needed to go to Heart n Soul to use their WiFi)
2
(Mark)	YES	NONE	NONE15	Sent iPad 2 with 3G	Realized his house had Wi-Fi connection so we exchanged the iPad for an iPad without 3G
3(Ifeoma)	NO (can access elsewhere)	Phone, iPad 2, laptop	email, WhatsApp	None needed	Started off using her phone, then iPad worked better
4
(Pino)	YES	Simple flip-phone	NONE15	Sent iPad pro, then iPad 2	Tried to use carer’s laptop, then iPad 2 worked best
5 (Michaela)	YES	Phone, desktop	email, WhatsApp	None needed	No issues
6(Lizzie)	YES	Phone	email, WhatsApp	Sent a new iPad 7	Started off using her phone, then iPad worked well
7
(Robyn)	YES	Phone, iPad 6, laptop	email, WhatsApp,Zoom	None needed	Uses laptop but if not well or tired then iPad
8 (Donald)	YES	Phone, iPad 5, laptop	email, WhatsApp	Sent a new iPad with 3G that had Zoom set-up already so he could easily join	Started using his own iPad and a laptop once familiar with Zoom meetings so we re-collected the iPad with 3G

Every co-researcher had very different levels of experience with and access to digital technologies. Therefore, access to digital technologies and the internet were monitored on a regular basis. Out of the eight co-researchers, seven had WiFi access at home. Two co-researchers had not used any digital devices before. Time was taken to explain the potential benefits of having an iPad during the COVID lockdown period. Three co-researchers used their own devices throughout.

4.2. Co-research workshops

4.2.1. Using Zoom

Video conferencing requires effective use of mute/unmute features, and participant’s faces need to be visible. By meeting no. 15, all co-researchers managed to perform these tasks independently and felt much more comfortable doing so. As a result, each research meeting ran smoothly. We were able to focus on survey data co-analysis as opposed to resolving technical difficulties. Although assessing the usability and accessibility of video conferencing is potentially very interesting, it is not the topic of this paper. However, it is important to note adapting to video conferencing with this group of co-researchers took time (see SupplementNo.4 for details).

4.2.2. Safeguarding

Co-researchers were potentially at risk to survey responses on LD conditions. To reduce risk and increase recovery time, the first four meetings were short - 1 hour in duration. Starting with meeting no. 6, we added the 15 min non-recorded section to the agenda. After meeting no.9, co-researchers and other team members became more confident. Meeting durations were extended and included two short breaks. When co-analyzing sensitive content, meetings needed to involve at least one senior member of Heart n Soul (excluding the facilitator). A second was available on request. During co-analysis of the open-ended question, a Zoom breakout room was used as a safe space where co-researchers could take a break or perform a creative activity with others from Heart n Soul. For every co-researcher with a support worker, we ensured they were involved at every stage of the study. In these cases, support workers were key for ensuring co-researchers could participate (see SupplementNo.4 for details).

4.3. Co-analysis of the demographic data (Figure 2)

Figure 2. Our journey to produce the co-designed bar charts. Step 3 represents the bar charts re-created by one of the co-researchers based on the ethnicity question. The co-researcher describes the process in detail in a video (See Supplement No.5-Ethnicity including all the individual images shown in Figure 2).

By the 15th of May 2020, 1,123 individuals fully completed the survey. The vast majority reported their gender as ‘female’ or ‘woman’ (76%) and their ethnicity as ‘white’ (84%), with the mean age of 41 years old. In addition, 81% completed a university level education (Bachelor’s Degree and above) and half were in full-time employment. Out of the 83% who lived in the UK, the most were from the Greater London area (36%). Also, 79% reported to know a person with LD, 27% had a disability or impairment and 38% reported mental health issues. Below, we report the full co-analysis process for the questions on ethnicity and the level of education.

4.3.1. Step 1 (Exploration)

Using the graphs created by the standard Stata commands for several demographic variables, it became apparent that co-researchers preferred a white background, black labels, a smaller number of categories, vertical bars with no gaps in between, and the categories presented in descending order (highest frequency for a category first). They also wanted graphs (‘tables’) to be more colorful. They also made clear the use of jargon was not welcome and simpler language should be used to explain graphs. Co-researchers were also introduced to simple pie charts. Co-researchers claimed to understand these well. One stated that pie charts “present perfect way of looking at something”. However, they stated an overall preference for bar charts on multiple occasions which is in accordance to the literature [10, 43] ¹⁶.

4.3.2. Step 2 (Improvement)

Following feedback, the improved graphs utilized the d3 palette¹⁷ with a categorical colour scheme (d3 20) to accommodate multiple categories. This made each bar more distinct and featured a legend inside the plot region to the top right. Hard copies of the graphs utilizing both percentages and frequencies were sent to the co-researchers. Initially, workshop leaders referred to these as ‘homework’. Co-researchers associated this term with negative school experiences. Workshop leaders then referred to them as ‘creative activity’ packs. This reflected the goals of the project more explicitly, including the co-researchers’ freedom to creatively express themselves as part of the research, and the idea that there were ‘no wrong answers’. Also, at this stage, the difference between percentages/proportions and frequencies were explored. Co-researchers elected to use frequencies for reasons summarized by Robyn: “Percentage is a very abstract and inaccessible way of explaining proportions. We should go with true numbers”. However, often these were the graphs of percentages/proportions that co-researchers selected for co-analysis and referred to in the meetings, as in the example of a redesign by one of the co-researcher’s shown below.

4.3.3. Step 3 (Redesign - Drawing)

Ethnicity (Step 3 in Figure 2)

Although the individual co-researchers had diverse embodied experience of ethnicities, we only co-analyzed 5 broad ethnic groups. One co-researcher became very interested in the ethnicity bar chart. Like many of the co-researchers, she enjoyed working with pen and paper because “they are more flexible…it takes me a very long time on the computer, but I can do it eventually, so I prefer using pen because it’s easier”. She first re-drew the graph trying to be very exact about the proportions of the bar heights to ‘get it right’. When asked for feedback in a Zoom meeting, the group appreciated the vibrancy of the selected colors. In a follow-up meeting (no.7), the Research Fellow encouraged the co-researcher to approach the re-design process more creatively, presenting several graphs¹⁸ made by data journalist, Mona Chalabi. The co-researcher was enthusiastic about the potential avenues for graph re-design, drawing people of different heights and cultural dress. She made clear that for her work to be successful, a summary note for every graph was required. At Zoom meeting no.13, she presented her re-deigned artistic bar chart to the team as a graph of ‘ethnic minorities for people living in different parts of the world’¹⁹ with the purpose to “be more creative and decorative which is clearer and more understanding” with the hope that “everyone should love it!”. Overall, the co-researchers were impressed with the outcome “that was brilliant!” and “was a great way to show your own understanding”, one confirmed that he “would find it hard to do that!”. Another co-researcher was pleased with the labels added: “when you put the names at the bottom, before I had troubles of understanding!” confirming that the legend did not work well for multiple co-researchers due to the constant need for non-accessible cross-referencing of the information. Following this positive feedback, the co-researcher agreed to record a video of her explaining the re-design process in her own words with no further guidance (See Supplement No.5-Ethnicity). It also became apparent that co-researchers found pie charts less accessible which is in accordance to other studies in the literature [10, 43] ²⁰.

In a Zoom meeting taking place on the 25th of June 2020, one month after George Floyd’s death and during the Black Lives Matter protests, co-researcher Lizzie, a woman of color, shared with us her interpretation of the ethnicity bar graph. Lizzie expressed shock at the small number of adults who identified themselves as ‘Black’, stating she felt some people of color may report they are ‘White’ because they are ashamed to report they are ‘Black’. During the conversation, she suggested to work with another co-researcher to engage more fully with Black communities as part of the project.

Level of education (Step 3)

This presented many challenges as not all co-researchers were familiar with post 16 qualifications used as response options (See SupplementNo.5-Education). Since 81% of the respondents completed a university level education, time was spent clarifying this category and evaluating its significance. Robyn re-designed the bar chart using plasticine and prepared a video recording (See SupplementNo.5-Education) in which she explained the advantage of presenting an ordinal variable according to its ordinal scale: “Bar graph does not communicate that education builds up”. Pino noted that the highly educated respondents may be “more open to experience” therefore he predicted the responses to be “friendly”. Pino was most interested in a respondents’ values, clarifying “if they trust us, it is a good person - a person with good manners, kind, generous, understanding… But we don’t know what kind of people they are so… I would like to know! I would like to know whether they believe in us or something else”. For David, highly educated respondents resembled students who “are doing research on them (people with LDs), get good grades for it, and then the university does not want to admit them (people with LDs) or give something back. It should be a mutual relationship…”. Although out of scope for this paper, co-researchers requested a Stata-derived bar chart depicting the full employment status by the level of education. We were surprised that between 33% and 47% of those with higher education were not in a full-time employment: “Just because you’ve gone to uni does not necessary mean you will be in full time employment” which, as one of the co-researchers pointed out, raised key questions; “Why is the job market not helping these qualified individuals? Education is very expensive and takes long time to pay back”, with the overall summary that “formal education does not lead to a great job, and just because you do not have education, does not mean you are stupid”.

4.3.4. Step 4 (Redesign - Stata)

Following feedback, Research Fellow redesigned the bar charts in Stata using the s1 color palette. This time, horizontal bars were used and the legend was removed with each bar of frequency being labelled and an emoji added on the left hand side of the graph. The horizontal pattern of the bars chosen to prepare co-researcher for upcoming LDAvis sessions. Hard copies were sent to co-researchers (See SupplementNo.2). To reinforce that respondents were only a small proportion of the population, and not representative of our communities, we adopted the phrase “the people who answered our questions” to describe them. Respondents were predominantly ‘white, young, highly-educated female’ which resembles the profile of the Guardian readership²¹.

4.4. Co-analysis of the STM output: “What do you see when you see me?”

At this point we considered all responses received up to the 21st of August 2020. The co-researchers selected the first co-designed question in the survey for the in-depth analysis: “Hi, my name is Lizzie. I’ve got a question for you… People stare at me all the time. What do you see when you see me?” because of “..the way Lizzie portrays herself and the drawings, it is very intense, these questions will stand up more to more people.”

Overall, 1,689 respondents completed Lizzie’s question including 100 selecting the ‘prefer not to say’ option. Out of the remaining 1,589 responses, 1,574 were in a text format, 11 using audio (7 transcribed), 2 using photo (all transcribed) and 2 using video (all transcribed). Considering the text responses with empty/non-meaningful fields and the 4 non-transcribed audio-responses, and after the data processing steps in R, we were left with a corpus of 1538 documents, 792 terms and 11192 tokens for the main topic modeling.

After STM analysis, the proportion of the corpus related to the most common topic was equivalent to 15.3% with the three highest probability words (three most common words) being “see”, “person”, “just” noting a significant drop in the estimated term frequency for the term “just”. Although the term “see” would be normally made redundant in response to Lizzie’s question “What do you see when you see me?”, we decided to preserve it as it became one of the significant features in the co-analysis process as further explored below. Initially, 10 representative responses were selected, but after careful consideration of the last 5, we were concerned with the content and what effect it may have on co-researchers at this stage, especially Lizzie.

4.4.1. Step 0 (Figure 3 Creative exploration)

Figure 3. The artist prepared a drawing summarising the key STM processing steps including ‘stemming’ and ‘the bag of words’ translated into the concept of ‘the basket o words’ for further exploration with co-researchers (Step 0).

Based on co-researchers’ interpretation of the STM model discussed in one of the co-research meetings, the artist prepared a drawing summarising the key STM processing steps including ‘stemming’ and ‘the bag of words’ (Figure 3). Here, explained by the artist in an accessible language, the survey responses from the co-designed survey go to a computer which counts the words that people have said. The computer recognises particular words that are used multiple times. For Lizzie’s question, it was discovered that people used three words a lot i.e. three versions of words a lot. The computer counts these words and divides them into separate baskets. All the words in each basket is counted up and they are all put into a graph where at the top is the word that is counted the most and it gradually gets less.

4.4.2. Step 1 (Figure 4 Three words)

Figure 4. Step 1. On the left, the three most common words presented to co-researchers in Jamboard along with several examples for ‘the baskets of words’ (bottom). On the right, the three most common words re-designed by a co-researcher along with a representation of a co-research meeting in our inclusive and accessible physical space (top).

At first, to generate the hypothesis, co-researchers - and Lizzie in particular - were asked “What do you hope that people would answer?” to clarify the co-researcher’s expectations. At first, Lizzie hoped that people would answer ‘a person’ which she elaborated on in the following way:

“OK, I see that she’s got LD but hang on - I don’t see any boundaries for what this person can do. And let’s see what this person can do. Instead of saying, she has a disability…Can you help us? Having more education about who we are and about changing your point of view on things and changing your mindset.”

Next, the top three associated words for Topic 1 were presented to co-researchers using sticky notes in Jamboard referring to the ‘baskets of words’ as examples to initiate conversation. This is how Lizzie interpreted the three words:

“What we see - the ‘person’. ‘Just’ - just accept it. See what the person is and just accept it!”

All co-researchers were then engaged in at times very emotional conversation, and throughout they recalled events that had significant impacts on their lives. Although several co-researchers were pleased with how people responded to Lizzie’s question “there was not such a thing in the past - people with disabilities never had a chance - it is positive move, a positive change”, one noted that “people should be left alone - that way they are themselves” and another added that “the reality is different” [to what the survey respondents have said].

Based on the material from the ‘creative pack’ for Steps 1 – 2 (See SupplementNo.3), Ifeoma depicted the three words and listed their possible versions in her drawing. In Ifeoma’s interpretation, she refers to ‘personality’ and the idea of ‘justifying’ as well as ‘justice’ in one of the images which one may interpret as looking beyond one’s disability and desire for justice. You can watch a video of Ifeoma in conversation with the artist in which Ifeoma elaborates on her artistic interpretation of the co-analysis process pertaining to the top three associated words in particular (Figure 4, on the right) along with the artist’s explanation of Figure 3 “to try and make the co-analysis process more accessible and easier for everyone to understand” (See SupplementNo.6). In the video, Ifeoma mentions that the artist’s interpretation allowed co-researchers to translate their ideas and do it differently, with more confidence: “You can look at something [the artist’s drawing] and then everyone has their own exploratory ideas”.

4.4.3. Step 2 (Figure 5 Word cloud)

Figure 5. Step 2. On the left, the word cloud for Topic 1 generated as part of the STM model. On the right, a creative exploration of ‘the baskets of words’ based on the words presented in the word cloud.

Informed by ‘the baskets of words’ concept explained in Step 0 and the material included in the ‘creative pack’ (See SupplementNo.3), Robyn creatively explored the words appearing in the word cloud “to put the words into pictures so that they would be a bit more accessible. And some of the words were quite hard to draw like ‘age’, for example”. For example, Robyn presents the possible versions of the word roots ‘pass’ as ‘passenger’, ‘passage’, ‘passive’, and ‘a pass’ thus appreciating the potential diversity of responses to and about Lizzie and therefore the diverse perspectives of the survey respondents.

Some words did not make too much sense and some sparked attention among the co-research team members. As depicted in her image, Robyn was curious about the term ‘plug’ appearing in the word cloud. After inspecting several of the survey responses, we noticed that the term ‘plug’ is used as ‘ear plugs’ supposedly referring to Lizzie’s ‘colorful’ hearing aids: “they don’t have a lot of information, they don’t have a lot of language – and language is power […] then people might be more considered of each other because they would have a bit more knowledge”. By recalling her own experiences of how the public perceives people with LDs and autistic people, Robyn suggested that out survey is like educating the public in a fun way – “if you give people the opportunity to look at us, so that people can get used to us […] people might be more open towards one another”.

The single word ‘age’ led to a long reflective discussion about the values of Heart n Soul and their relevance to current society. Robyn highlighted “I don’t think about peoples’ age, that surprises me […] because age is irrelevant […] I guess maybe the wide world thinks about age a lot more than I do”. The team agreed that at Heart n Soul people are not judged by their appearance, identities or disabilities. Another co-research, David, explained the co-analysis of the word cloud by providing a hypothetical example inspired by the emerging discussion about the power of the survey and how it invites people to reflect on their lives:

“[we are] trying to find out the key words, what people say – like POWER – when they say ‘power’, it could be the power of people and the power of what disability they gonna go through. Or the power that people don’t get if they’ve got a disability. Because sometimes we don’t get listened to… But you just have to try it!”

4.4.4. Step 3 (Figure 6 Most representative responses)

Figure 6. Step 3. On the left, 5 (out of 10 derived) most representative responses of Topic 1 were discussed with the co-researchers. Following Lizzie’s advice, here we report only the ‘positive’ parts of the responses (3rd and 5th are partial). On the right, an application of Step 3 in a new creative work from one of the co-researchers.

Next, 5 (out of 10 derived) most representative responses of the most common topic were discussed with the co-researchers (Figure 6). Importantly, at Lizzie’s request, some ‘negative’ responses were removed so that only ‘positive’ answers remained. This was challenged by another co-researcher who stated, “we will be deemed not scientific enough”. Data-wise, this was a significant challenge, as Lizzie’s opinion and the group’s collective well-being was uppermost. During this phase, most co-researchers focused mainly on ‘negative’ responses, with Lizzie stating, “I understand it and I don’t”. One of the respondents who provided a ‘negative’ response was described by Robyn in the following way: “this person is honest. You do not know ways to describe disabled people so maybe people just do not have a lot of words about disability that they know” and “Maybe I don’t look the same as other people but I am who I am. [there are] people who want to fit in, those who want to be the same as everybody else. I don’t think it is a bad thing, they are just being honest”. Therefore, our discussion oscillated between concepts of honesty and trust: allowing respondents to be honest in their answers and having trust in our research goals and each other. This idea of being able to confidently face and discuss the more ‘negative’ responses in our co-research meetings was a total breakthrough to all of us in the co-analysis process.

Michaela, one of the co-researchers, took Step 3 in the co-analysis process to the next level (Figure 6). In her painting, by depicting a different person asking the same question “What do you see when you see me?”, Michaela naturally noted that had there been a different person asking Lizzie’s question, peoples’ perceptions and the survey responses would have been different with some common elements: “Image could be anybody. Whoever looks at it, can see a person etc. What if we had her drawing in the survey? People would say something different”. In here, and throughout the entire co-analysis process, co-researchers anticipated the friendlier and more understanding ‘positive’ survey responses.

4.4.5. Step 4 (LDAvis package)

Results from LDAviz were then presented (See SupplementNo.7). Academic researchers attempted to explain the modelling process, and the meaning of the visualization. A number of issues were raised. The font size was too small and after enlarging it by 300%, it was difficult to see which elements it referred to. Some comments by co-researchers were that ‘they couldn’t see it’, ‘the words were difficult to read’, ‘the font size too small’, and overall, it felt ‘too confusing’, ‘too abstract’, ‘hard to understand’ and ‘disengaging’ and “too much information!”. The intertopic distance map outcomes on the left were very difficult to cross-reference with the most relevant terms on the right. The group’s attention was lost, with one co-researcher falling asleep during the session. Robyn then made a list of the most common words using an Excel sheet with an enlarged font size as an alternative approach.

4.4.6. Step 5 (Evaluation)

Through arts-based approaches, an underlying theme for the topic was explored which reflected co-researchers’ new ways of understanding the public, themselves, and their place in the world. In Step 5, towards the end of the co-analysis process, the team produced a video²² in which the co-research team members explain “How we made sense of our data” including their understanding of the entire co-analysis proces²³, motivations and proud moments.

Nowhere and Everywhere: In the video, Robyn defines co-analysis from the inclusive research perspective as “when a group of people all equally take part in working together to understand data and understand what it means”. Ifeoma recalls that “she did not find it [co-analysis] easy at first” but she was motivated by the idea of making the numerical graphs more accessible using creative methods and her artistic skills. Lizzie and Pino also explain that every co-researcher had a different role and was free to contribute in different creative forms, even with song-writing and performing music expressing not only what they did, but how they felt throughout the co-analysis process. Co-researchers enjoyed co-analysis because of the opportunity to meet regularly and to make research accessible to everyone through the right support and respect. As Robyn mentioned, everybody was treated as equals so “we democratized research for people with LDs and autistic people”. This positive and confident feedback reflects co-researchers “growing in confidence” throughout the co-analysis journey and is best summarized by David’s remarks: “We are making a difference because we are the first to do it so we set standards for others”. By equally involving co-researchers with LDs in co-analysis, who often report feeling lonely and isolated [6], co-researchers meaningfully connected with the public in the digital world thus transitioned from “Nowhere to Everywhere”.

Change Though Design: As presented in the video, the co-analysis process has challenged our assumptions and the survey responses gave us the opportunity for reflection as to how we have worked, why, and who we are as researchers and members of the public. By providing the public with a safe space for an honest conversation in a digital environment, we understood that many people are keen to have a conversation with us and want to know more about us. Co-researchers began to appreciate that to change the public, they need to change themselves in the first place. Lizzie elaborates on this concept, and the co-analysis journey, in her song which she prepared together with Pino and Mark S (Supplement No.8).

We analyzed and studied what people have said

We’ve made graphs, we’ve made bubble writing – some of it doesn’t make sense

We’ve put them into baskets – all the words sound the same

We’ve put them into words but only one theme – and the words just don’t make sense

But we put them into sentences and it makes this perfect sense

The words that you’ve been saying has really blown our minds

So what we need to do is sit and study and analyze

Now we really know how you people really feel

Why didn’t you tell us is the first place so then we could be real?

[…]

We’ve been working ever so hard during the lockdown

Seeing all these words onto paper

Now look what we have found!

We’ve analyzed them, we want to know how you really feel

Thank you for answering our questions!

[…]

We are working with scientists and we are coresearching

We know what we are doing!

We gonna change peoples’ minds

And we gonna change what we’re doing

[…]

How we can change peoples’ minds?

How we gonna change our minds?

When we worked in this we didn’t really know how we could change our lives today

When we worked in this we didn’t really know how we could turn it around again

5. Discussion

In summary, we found co-researchers gained confidence from creatively re-designing more standard numerical information first before moving on to more complex tasks. We derived an efficient five-steps STM co-analysis process for creative, inclusive, and critical engagement of data by co-researchers. Overall, it was sufficient to co-analyze the highest probability words for the most common topic for creative, inclusive, and critical engagement of co-researchers with the data. LDAviz was considered very inaccessible by co-researchers due to having very small text and a high number of visual elements. Co-researchers observed that by trying to understand and impact public opinion, their own perspectives also changed. At first, co-researchers were apprehensive of people’s responses based on their lived experiences of LDs and/or autism. Gradually, step by step in the co-analysis process, they gained confidence in their analytical skills. This also helped co-researchers reach conclusions about ‘negative responses’ - for example, seeing them as being indicative of respondents lack of knowledge of LDs. This reinforced the perspective that educating and engaging with the public in these ways was worthwhile. Further, our study demonstrated that data science approaches applied in the context of co-research can be a fruitful and useful method broadening both quality and applications of data science, placing lived experience at the core of data science processes and methods.

Framing and challenging normative approaches: We subscribe to a broader emerging consensus in HCI that stands for the complex relevance and nuanced internal validity of participatory design (PD) processes [5]. Yet, the originality of our project has been precisely the choice of using quantitative methods and combining them with PD, thus bridging the gap between the positivistic and the post-modern paradigms, positioning ourselves beyond traditional academic and epistemological grievances. Our co-research with people who are often marginalized and systematically ignored, embodies the more critical views of science and AI and is an example of AI ‘decolonization’ viewed from an engagement perspective [18] by providing the grounds for democratic, accessible and inclusive ML pathways (ICML 2020). Mahr and Dickel (2019) [15] report that “Dominant forms of contemporary big-data based digital citizen science do not question the institutional divide between qualified experts and lay-persons”. Our study challenges this institutional divide and “the system of professional science” on multiple levels starting with the co-design of an accessible survey system and the creation of an online survey by co-researchers with lived experiences of LDs.

Contributions to data science approaches: To challenge the current status quo in data science and AI, co-researchers “had to play people [academics] at their own game” (Robyn). This is why co-researchers with LDs worked with numerical data to be able to critique it and were encouraged to engage with data exploration creatively to understand key findings and initiated the discussion of scientific bias. Also, survey respondents were free to respond to the co-designed open-ended questions using their own frame of reference [38] “even if this might seem inappropriate or ‘irrational’ to the survey designer or analyst” [33]. Most significantly, we show that by utilizing machine learning-based models to derive topics and themes, our team members undertook co-analysis on approximately ‘equal grounds’. This had the effect of diffusing power differentials, which is at the heart of democratic research practice and co-produced research [8].

The study has several limitations, not least because we test several novel approaches all at once. Some may question the involvement and contributions of non-academic researchers with lived experiences of LDs in the co-analysis of the STM outputs. However, ML models have been misinterpreted by data scientists on multiple occasions leading to discriminatory outcomes [9]. The authors explain that ML interpretability techniques entail a variety of inherently biased approaches, which we think could be potentially extrapolated to the visualisation tools such as LDAvis. Further, as our duty has been to protect the well-being of co-researchers, we were compelled to co-analyse only the first 5 of the 10 most representative responses for the most common STM-derived topic. This complied with co-researcher Lizzie’s request for ‘only the good and more positive answers’ to be reported in detail. This problematizes and challenges the balance between ‘rigorous science’ and safeguarding principles within ‘inclusive research’, but also makes clear that many negative responses were received and revealed by the STM process. Further, towards the end of our co-analysis, several co-researchers raised issues of potential “self-selection” bias, realising that had they teamed-up with different people they could have viewed the data from a different perspective.

Conclusions and Future work: Altogether, we conclude it is not only possible to operate on multiple layers of analysis, using abstract concepts whilst generating different outputs while prioritizing well-being of all team members, but it is legitimate as rigorous scientific evidence within the scope of inclusive collaborative/participatory research approaches.

Overall, the co-analysis of demographics acted as a positive prompt for hypothesizing what respondents might say in open-ended questions, whilst providing co-researchers with confidence to evaluate more complex data. Also, STM modelling provided insights to the research team and facilitated co-research through by aiding the diffusion of power differentials. However, visualisation tools were largely insufficient to meet co-researcher’s accessibility requirements, although we recognize the situation might have been different had co-analysis taken place in a physical space with a large monitor or projection. As future work, making data analysis programs and software more accessible will enable more non-academically trained individuals with lived experiences to actively shape research and may in turn impact on issues of trust with respect to data scientists and ML interpretability tools [9]. Per Lizzie’s feedback, future studies of this kind should make the extra effort to involve non-verbal or hard of hearing individuals as well as those with visual impairments in the co-research process.

Even though there were several points in the course of the study when “it wasn’t easy” [31] and we were unable to have all the co-researchers at once in a single Zoom session (at least one co-researcher not present), there was a general feeling of accomplishment which is not only precious to all of us on a personal level but in line with the goals of the second generation of inclusive research [10, 22].