In-Motion-App for remote General Movement Assessment: a multi-site observational study

Abstract

Objectives

To determine whether videos taken by parents of their infants’ spontaneous movements were in accordance with required standards in the In-Motion-App, and whether the videos could be remotely scored by a trained General Movement Assessment (GMA) observer. Additionally, to assess the feasibility of using home-based video recordings for automated tracking of spontaneous movements, and to examine parents’ perceptions and experiences of taking videos in their homes.

Design

The study was a multi-centre prospective observational study.

Setting

Parents/families of high-risk infants in tertiary care follow-up programmes in Norway, Denmark and Belgium.

Methods

Parents/families were asked to video record their baby in accordance with the In-Motion standards which were based on published GMA criteria and criteria covering lighting and stability of smartphone. Videos were evaluated as GMA ‘scorable’ or ‘non-scorable’ based on predefined criteria. The accuracy of a 7-point body tracker software was compared with manually annotated body key points. Parents were surveyed about the In-Motion-App information and clarity.

Participants

The sample comprised 86 parents/families of high-risk infants.

Results

The 86 parent/families returned 130 videos, and 121 (96%) of them were in accordance with the requirements for GMA assessment. The 7-point body tracker software detected more than 80% of body key point positions correctly. Most families found the instructions for filming their baby easy to follow, and more than 90% reported that they did not become more worried about their child’s development through using the instructions.

Conclusions

This study reveals that a short instructional video enabled parents to video record their infant’s spontaneous movements in compliance with the standards required for remote GMA. Further, an accurate automated body point software detecting infant body landmarks in smartphone videos will facilitate clinical and research use soon. Home-based video recordings could be performed without worrying parents about their child’s development.

Trials registration number

NCT03409978.

Strengths and limitations of this study.

• A cohort of families of high-risk infants frequently seen in neonatal infant care units follow-up settings.

• In-Motion-App standards for remote General Movement Assessment communicated through a simple and short animated video.

• Data from a motion tracking software on smartphone videos pioneering automatic and markerless infant motion capture.

• Study did not assess sociodemographic factors as reasons for families not to record or return videos.

• Study did not evaluate how markerless infant motion capture on smartphone videos can be used for prediction of cerebral palsy outcome.

Introduction

Cerebral Palsy (CP) is the most common physical disability in childhood. Diagnosis is typically set between 12 and 24 months corrected age.^1–3 Early developmental screening of high-risk infants to predict future neurological impairments is today a priority for clinicians and researchers, and most parents express interest in such neurodevelopmental screening.^{3 4}

The General Movement Assessment (GMA) has been recommended in combination with MRI to achieve a CP diagnosis before 6 months corrected age in infants with newborn-detectable risk factors.³ Early detection of CP has the potential to improve the organisation and resources used in follow-up screening at hospitals and reduce medical complications for children with CP. The fidgety type of general movements (GMs) observed at 9–20 weeks’ corrected age has shown the highest predictive validity for later CP, compared with the writhing type of GMs observed before 9 weeks corrected age.^5–8 Video recordings for GMA must follow requirements for infant state, position and clothing and are scored by certified and trained assessors.⁹ Such trained GMA observers also have the expertise needed to ensure that video recordings fulfil the requirements for a valid GMA.⁹

Access to trained observers using the GMA in hospital-based follow-up programmes is limited by geographical constraints and lack of GMA expertise.¹⁰ As healthcare and parents move into the digital age using smartphones to share videos via internet, opportunities to perform remote GMA have developed. Recently, the Baby Moves smartphone app was presented for remote GMA within research settings.^{10 11} However, smartphone apps for health data capturing in clinical settings are rarely assessed and usability tested.¹² To be feasible in a clinical follow-up setting, home-based video recordings must fulfil basic GMA requirements without the need for comprehensive parental training or guiding.

Video recordings by hand-held smartphones introduce movement artefacts in the camera. Computer-based methods for objective detection of infant GMs^{13 14} may be jeopardised by such artefacts. Our research group has recently presented a machine-learning model which predicted CP with high accuracy (sensitivity of 92%, specificity of 81%) performed by clinician using a stationary camera, comparable to observational GMA.¹⁵ Important shortcomings of the method were the need for manual and time-consuming body point annotations, as well as the need for a stationary camera. Hence, an automated 7-point body tracker has been developed by our group and needs to be validated on recordings taken with hand-held cameras.

Provided that GMs can be assessed, and computer-based infant body point tracking can be performed on videos taken with a hand-held camera, it is possible to perform remote GMA as well as automated infant body point tracking for a computer-based model for the prediction of CP. Thus, the In-Motion instructional video has been developed so that parents can perform home-based videos with quality standards feasible for remote GMA and automated infant body point tracking. Feasibility of the In-Motion instructional video was assessed in a multi-site study including families of high-risk infants from Norway, Denmark and Belgium.

The main aims of the study are as follows: (1) to determine whether videos taken by parents with a hand-held camera were in accordance with the standards set in the instructional video, and whether the videos could be scored by a trained GMA observer; (2) to assess the accuracy of a 7-point body tracker software based on the same recordings; (3) to describe parents’ perceptions of the instructional video and filming their baby in a home environment.

Methods

Design

Multi-centre prospective observational study.

Patient and public involvement

The study protocol including the parental survey content was developed and designed collaboratively with representatives from The Norwegian Cerebral Palsy organisation and The Norwegian Premature Association.

Participants

Parents of infants admitted to one of five participating level III–IV neonatal infant care units (NICU) in Norway (3 hospitals including 13, 11 and 4 families, respectively), Denmark (1 hospital including 43 families) and Belgium (1 hospital including 15 families) from 2018 to 2019 (12 months recruitment period) were consecutively recruited at referral to the hospital follow-up programme before discharge from the NICU. Families were recruited based on willingness to participate and the infant being evaluated as at high-risk of CP. In Norway and Denmark inclusion criteria were (1) birth weight (BW) ≤1000 g (extremely low BW and/or gestational age (GA) <28 (extremely low GA), (2) neonatal arterial ischaemic stroke, (3) neonatal encephalopathy, (4) other significant risk factors. In Belgium, only infants with GA <32 weeks or with perinatal stroke were included.

Data collection procedure

Included participants were assisted by a research physiotherapist/paediatrician at the time of inclusion to download and instal the In-Motion-App by smartphone, containing the instructional video from Google Play or iTunes. They could ask any questions about the app and how to manage the software. They got information about the time window for performing two separate video recordings for their infant between 12⁺¹–13⁺⁶ and 14⁺¹–17⁺⁶ weeks post-term age (PTA).¹¹ The time points were defined to ensure GM videos from the fidgety movement’s period. If no videos were returned from the families before 17⁺⁶ weeks PTA, the local study coordinator contacted the family by phone to ask the reason why they had not uploaded any videos. The app was linked to a secure online server hosted at St. Olavs Hospital in Trondheim, Norway, and was available for i-operating system and Android. After the end of the second time window, the families were contacted by email with a link to an online Norwegian University of Science and Technology (NYNU) survey to collect information about their opinions using the app and the In-Motion instructional video.

In-Motion-App and instructional video

The In-Motion-App and instructional video was designed by GMA trained personnel (LA, TF, RS, SO) at St. Olavs Hospital in Trondheim, Norway, for parents to give basic insight into recording standards needed for GMA and lighting and stability of camera. It was made as a short animation with simple drawn sequences containing a minimum of text. The instructional video was deployed to the parents by downloading the In-Motion-App developed for this study, and videos could be uploaded to be remotely assessed by a trained GMA observer.

After downloading the app and getting basic information from the local study coordinator, parents logged into the app with a username and a password. They typed in the first name of their child and the expected date of delivery (due date). The In-Motion-App generated two separate time windows between 12⁺¹–13 ⁺⁶ and 14⁺¹–17⁺⁶ weeks PTA¹¹ and visualised them in a graphical timeline to show when videos should be taken. A red dot illustrated today’s date placed on the timeline, helping parents to plan when to perform the video recordings. In addition, a pop-up message reminded parents to prepare for videos a week before the beginning of each time window.¹¹ The In-Motion-App was constructed in such a way that the video recording automatically stopped after 3 min and asked the parents whether to upload the video or not.

The instructional video was 2 min and 47 s long. Before taking videos, parents were told to look through the In-Motion instructional video which was available from the app menu. They could watch the video as many times as they wanted until they felt confident performing the recording. The main themes aimed at ensuring quality standards for remote GMA included: (1) clothing of infant (just a diaper or a onesie), (2) surface/underlay for infant (single-colour blanket or rug), (3) lighting (enough light avoiding sidelight that can cause shadows), (4) state of infant (awake, alert, content, not disturbing baby, no pacifier), (5) positioning (baby on floor—stand next to the baby’s feet, whole body must be visible) and (6) length of video (3 min). In addition, instructions were provided about how to keep the smartphone steady and ensure that the whole infant body was observable in the video image. Examples of some of the In-Motion instructional video themes are shown in figure 1. Parents were asked to consecutively upload videos to the server at St. Olavs Hospital in Norway.

Figure 1.

Screenshot of In-Motion instructional video showing examples of information about infant state, lighting and positioning of baby and person filming the baby.

Assessment of video quality for remote GMA

Videos were assessed by a certified GMA observer with respect to the following standards⁹: (1) GMA standards: active movements (not hypokinetic), supine position, correct state, adequate clothing (diaper or a onesie), no disturbances during recording. (2) Additional In-Motion standards: adequate light, whole body visible, feet of parent visible in video (ensuring correct position of smartphone camera, see picture to the right in figure 1) and camera stability. Based on these standards, a classification was made by the same certified GMA observer as either ‘GMA scorable’ if all standard criteria were fulfilled or ‘GMA non-scorable’ if one or more standard criteria were inadequate. In addition, all videos were observed by the same GMA expert who also categorised yes/no whether the hand-held video had optimal stability, events of abrupt displacement, was predominantly unstable, whether an adequate underlay was used (firm, comfortable, large enough) and whether overall video image quality was sufficient (blurred/very blurred).

Assessment of GMs

All videos classified as ‘GMA scorable’ were consecutively assessed by one certified and experienced GMA observer that had passed advanced GMs courses under the General Movement Trust (LA). The use of one observer was chosen due to the study design not focusing on GMA and prediction of outcome. The observer had no knowledge about the infant’s clinical history. According to Prechtl’s method of assessment of GMs,⁹ fidgety movements were classified as continuous (FM++), intermittent (FM+), sporadic (FM+/−), abnormal (Fa) or absent (FM−).

In-Motion body point tracking

The infant motion tracker algorithm consists of a convolutional neural network trained on 7-body points on 14 900 video frames on high-risk infants that had participated in another study from our group.⁸ For further technical details of the previous trained convolutional neural net, the reader is referred to Groos and Aurlien.¹⁶ To evaluate the infant motion tracker, 5493 video frames was selected from a subset of 66 videos from 36 infants, recorded by the In-Motion-App by 19 September 2018. Eighty per cent out of the selected 5493 frames were selected by random. The other 20% were selected manually in order to include body part occlusions (eg, right wrist occluded behind left wrist) that may be challenging to track. The performance of the infant motion tracker was assessed and reported by the following three steps: first, the automatic motion tracking was performed to detect the position of 7-body points (nose, thorax (centre between shoulders), wrists, pelvis and ankles) in each of the 5493 video frames. Second, all 7-body points in the 5493 selected video frames was manually annotated. These manually annotations are the ground truth for the evaluation of the infant motion tracker. Third, the performance of the infant motion tracker is reported as percentage of points within a circular area centred at the manually annotated body point for the 5493 frames. In accordance with the established metric for evaluating pose-estimation,¹⁷ radius of the circular area was set to 10% of the infant head size and was normalised to adjust for different scaling (ie, video zoom) (figure 2).

Figure 2.

Accuracy of 7 estimated body points compared with manually annotated images. From left: table with proportions of correct detected body point; illustration of the computer-based detections according to 7-body points; the distribution of the left wrist body point detections (blue dots) relative to the manually annotated landmark (black dot) where 10% of the infant head size is used as a threshold (black circle).

Survey

Parents’ opinions about the In-Motion-App and instructional video were collected using SelectSurvey V.4 software (ClassApps, www.classapps.com) in Norway, operated by NTNU. In Denmark, parents’ opinions were collected using Smart Trails data management software (MEDEI ApS, www.medei.dk) and in Belgium the survey was collected by post. The survey was sent to the families by a link in an email within 1 week after the last video was returned. The survey questions were customised for the In-Motion-App based on a tool developed by Jin and Kim¹⁸ and a survey used in a similar study on the Baby Moves App.¹¹ It contained questions based on forced-choice questions covering the themes: (1) In-Motion-App, (2) In-Motion standards for remote GMA and (3) parental worries. A statement was made, and the parents indicated agreement or disagreement with the statement on a 5-point scale (online supplemental appendix 1).

Data analysis

Data were analysed using SPSS statistics V.26.0 (IBM SPSS Statistics). The data are presented as numbers with proportion (%) or mean with SD and range. Differences between infants with returned and no returned videos were analysed using the Mann-Whitney U test for continuous variables and χ² test and Fisher’s exact test for dichotomous variables. The accuracy of the 7-point body tracker was presented as the distribution of body point detections relative to the manually annotated body points, where 10% of the infant head size was used as a threshold.

Results

In total, 86 infants/families were recruited and 17 (19.8%) out of them did not submit any video (figure 3), leaving 69 families with 130 videos for analysis.

Figure 3.

Flow chart of infant/families with reasons for non-upload of videos. GMA, General Movement Assessment.

Twenty-eight (32.6%) families were included from three different hospitals in Norway, 43 (50%) from one hospital in Copenhagen, Denmark and 15 (17.4%) from one hospital in Gent, Belgium. Infant/family characteristics are shown in table 1.

Table 1.

Summary of infant/family characteristics

	Total N (%) (n=86)	N (%) video responders (n=69)	N (%) no video responders (n=17)	P value
Demographics
Boys, n (%)	51 (59.3)	42 (60.9)	9 (52.9)	0.32
Birth weight (BW), mean (SD), g	1952 (1107)	1915 (1124)	2105 (1055)	0.67
Gestational age (GA), mean (SD), weeks	32.4 (5.3)	32.3 (5.4)	33.4 (4.7)	0.43
Risk group
BW ≤1000 g and/or GA <28	25 (29.1)	20 (29)	5 (29.4)	0.77
Neonatal arterial ischaemic stroke	11 (12.8)	4 (5.8)	7 (41.2)	0.001
Hypoxic ischaemic encephalopathy	20 (23.3)	17 (24.6)	3 (17.6)	0.06
Others	32 (37.2)	27 (39.1)	5 (29.4)	0.38
		Infant families (n=63)
Sociodemographic data
Mother relation (survey), n (%)		48 (76.2)
Married/cohabitant family, n (%)		59 (93.7)
Age mother/farther, mean (SD, range)		31.8 (5.5, 21–6)
Age farther, mean (SD, range)		33.9 (6.9, 22–59)
Single child, n (%)		32 (51.6)
iOS vs Android
iOS users, n (%)		41 (65.1)

iOS, i-operating system.

Video recordings

Two (1.5%) out of 130 videos were shorter than 3 min (1 min and 12 s and 2 min and 17 s). The mean PTA at video recording was 14.5 weeks (SD 2.28, range 8.1–23.6 weeks).

Fifteen (11.7%) families returned one recording, 49 (71%) two recordings, 4 (5.8%) three recordings and 1 (1.4%) six recordings. One-hundred and seventeen (90%) videos were returned within the expected time window between 12⁺¹ and 17⁺⁶ weeks PTA. Two (1.5%) families returned two and three videos, respectively, which were all taken outside the time window (week 8, 10, and 21 and 23, respectively). Eleven families (8.5%) with videos from within the requested time window, had additional videos taken outside the expected time windows. Six videos from one family were taken at two different days; four videos in weeks 12 and two videos in week 14 PTA.

Remote GMA

Among the six videos returned from one family, the first one from each of the 2 days was selected for GMA analysis. Exclusion of the remaining four videos and additional four videos excluded due to PTA outside of the age required for assessment of FMs gave a total of 122 videos available for quality assessment.

One hundred and twenty-one (99%) out of 122 videos which were returned within the required time window were classified as GMA scorable. The video that was non-scorable had infant in side lying position. Details about compliance to the In-Motion standards are shown in table 2.

Table 2.

Compliance to In-Motion standards (n=126)

	Active movements (not hypokinetic)	Supine position	Correct state	No disturbances during recording	Adequate clothing	Adequate light	Whole body visible	Feet of parents visible
N (%)	126 (100)	125 (99.2)	122 (96.8)	124 (98.4)	125 (99.2)	124 (98.4)	124 (98.4)	116 (92.1)*
		Optimal stability	Abrupt displacement	Predominantly unstable	Correct base of support		Image quality
						Clear	Blurred	Very blurred
N (%)		80 (63.5)	26 (20.6)	22 (17.5)	119 (94.4)	114 (90.5)	11 (8.7)	1 (0.8)

*Three missing data.

General Movement Assessment

Of the 121 videos classified as GMA scorable, 3 (2.4%) videos were classified with exaggerated (Fa), 3 (2.4%) with absent (FM−) and 7 (5.6%) with sporadic (FM+/−) FMs. Eighty-seven (69%) and 21 (16.7%) videos were classified with intermittent (FM+) and continuous (FM++) FMs, respectively.

Computer-based body point tracking

The proportion of correctly predicted left wrist key point from 5493 tested video images was 83.15%. Details of accuracy of 7-predicted body points and mean value for all points are shown in figure 2.

Parent responses

Survey responses were received from 64 (92.8%) families of the 69 families who returned at least one video. Fifty-four (84.3%) of them observed the instructional video one or two times before filming their baby. No families returned a video without training on filming their baby first.

The majority of the survey respondents found the In-Motion-App easy to use. All respondents agreed or strongly agreed that it was easy to understand how to stand and hold the smartphone during the filming. Details about family responses are shown in table 3. Fifty-seven (90.5%) families strongly disagreed, disagreed or neither disagreed nor agreed that they did become more worried about their child’s development through using the In-Motion instructions.

Table 3.

Parents’ responses to the In-Motion-App and instructional video

	Strongly agree (%)	Agree (%)	Neither agree nor disagree (%)	Disagree (%)	Strongly disagree (%)
In-Motion-App
The In-Motion-App was generally easy to use	58.7	33.3	3.2	4.8	0.0
It was easy to enter the information needed in the In-Motion-App	46.0	41.3	11.1	1.6	0.0
The reminders about when the child should be filmed were helpful	44.4	41.3	6.3	6.3	1.6
The number of reminders about when the child should be filmed was suitable	25.4	39.7	15.9	14.3	4.8
There were no technical problems with uploading and sending the videos	55.6	22.2	6.3	4.8	11.1
In-Motion standards for remote GMA
It was easy to understand how I should stand and hold the telephone during the filming	60.0	40.0	0.0	0.0	0.0
It was easy to keep the telephone still while I was filming	16.7	46.7	30.0	6.7	0.0
It was easy to do the filming without disturbing the child	36.5	42.9	9.5	11.1	0.0
It was easy to understand how my child should be dressed when filmed	65.1	33.3	0.0	1.6	0.0
It was easy to understand how my child should be positioned and how the mat should be when I was going to film	63.5	28.6	4.8	3.2	0.0
It was easy to follow the instructions about how the lighting should be during the filming	36.5	38.1	14.3	11.1	0.0
Filming my child for 3 min went smoothly	39.7	39.7	19.0	1.6	0.0
Parental worries
I felt safe about uploading video of my child	63.5	27.0	7.9	1.6	0.0
I became more worried about my child’s development through using the In-Motion-App	1.6	7.9	28.6	22.2	39.7
Using the In-Motion-App made me more attentive to my child’s development	7.9	36.5	46.0	4.8	4.8

GMA, General Movement Assessment.

Discussion

In this study, more than 95% of families with high-risk infants filming their baby at home, returned at least one video that was in accordance with the In-Motion standards for remote GMA. Most families found the In-Motion-App easy to use and the instructions for filming easy to follow, and less than 10% of respondents became worried through using the In-Motion-App. Despite the use of hand-held smartphones introducing movement artefacts in the video image, our computer-based 7-point body tracker detected positions of the body points with high accuracy. To the best of our knowledge, this is the first automatic infant body point tracker that is tested on video recordings from hand-held smartphones.

This study has several strengths. First, it included families of high-risk infants frequently seen in NICU follow-up settings. We argue that this makes our findings robust and generalisable to comparable clinical settings. Second, the communication of In-Motion standards through a simple and short animated video, makes the instructions easily applicable to a broad range of different clinical settings. Furthermore, the accuracy of the 7-point body tracker software with the use of smartphone videos makes it pioneering in the field of automatic and markerless infant motion capture compared with other studies.^{19 20} This facilitates further development of methods for early automated detection of CP based on smartphone videos. Finally, the design of the study using an experienced and certified GMA observer for evaluation of video quality and a survey with very high response rate, makes the study quality high and the results trustworthy.

There are also several limitations. First, almost 20% of the included families did not return any video. This study was not designed to evaluate reasons for not recording or returning videos. We can, therefore, only conclude on the quality of returned videos. The questions in our survey may also have limitations, mainly covering topics favourable to participants returning videos, participating in follow-up and smartphone usage, giving little or reduced information about responders with low mobile health technology usage. Hence, problems encountered by families who did not record or return any video need to be further explored. Our findings are in accordance with the study by Kwong et al,¹¹ where 24% of families did not return any video using the Baby Moves App. These findings indicate that home-based video recordings for remote GMA is not a solution for all families and that it might be difficult in a clinical setting to know beforehand which families will return a video or not. Furthermore, 13 (19%) families returned one or several videos outside the required time window needed for a valid GMA. More than 90% of respondents found the reminders in the app helpful, but one-fifth disagreed that the number of reminders were appropriate. These findings indicate a limitation in the design of the app reminders and lack of programmed filming windows parameters. These app functionalities need to be improved in a process involving the users.

Second, our study comprised five different hospital sites in three different countries, and information provided to families when downloading the app may have differed. Additionally, there is a risk that the research personnel could have given more information and assistance to families than will be common in an ordinary clinical setting. Our study setting could therefore be in slight contrast to an ordinary clinical setting, where adapted and flexible family information is needed due to challenges in reduced participation in neurodevelopmental follow-up²¹ and racial and socioeconomic differences in mobile health technology usage.²²

Third, even though almost 20% of the hand-held videos in our study were classified as predominantly unstable, the GMA expert considered the videos to be GMA scorable. There is a risk that more videos might have been classified as non-scorable by another GMA observer with less training and experience or if there had been several GMA experts observing the same videos. To the best of our knowledge, there is only one study protocol planning to assess the predictive validity of GMA from videos taken with hand-held smartphones for CP outcome.¹⁰ Hence, further studies on the use of smartphones for GMA are needed.

Finally, our computer-based 7-point body tracker showed accurate estimations compared with manual annotations on the video image. However, further studies must explore how selection of body points, tracked body point accuracy and movement artefacts in camera will influence a machine-learning model for prediction of CP from smartphone video recordings.

This study facilitates and contributes to the use of smartphone technology for video recordings and remote GMA. Consequently, it will contribute to giving high-risk infants and their families equal access to GMA as an accurate method for early identification of CP, without geographical constraints. The use of early remote medical assessment will improve the organisation and resources used in follow-up screening at hospitals and have the potential to reduce medical complications for children with CP due to early detection. A clinical feasible computer-based movement analysis with equal accuracy as GMA, will greatly reduce the need for specialised GMA observers and provide an innovative resource-effective diagnostic measure.