Do consumers and healthcare professionals report the same adverse event differently? A paired analysis of duplicate vaccine safety reports in Norway

Abstract

Aim

This study aimed to compare how Norwegian healthcare professionals (HCPs) and consumers reported the same adverse event (AE) following immunization with vaccines against COVID‐19 during the COVID‐19 pandemic. Specifically, we aimed to compare the extent to which HCPs and consumers reported information relevant for assessing the causal relationship between the AE and the suspected COVID‐19 vaccine, with a particular focus on the type of unstructured information provided by each group.

Methods

Verified duplicate pairs of AE reports following COVID‐19 vaccination were extracted from the Norwegian Adverse Drug Reaction Registry. Free‐text case narratives were analysed using thematic analysis to categorize types of unstructured information. A scoring system quantified completeness of causality‐relevant information, with final scores categorized into levels of completeness. Time intervals from AE onset to reporting were also calculated.

Nineteen categories of unstructured information were identified. HCPs more often provided vaccine dose details, clinical observations, test results and causality reflections, while consumers more often described dietary supplements, life consequences and subjective AE explanations (p < .05). In 85% of 127 duplicate pairs, HCPs and consumers provided at least 50% complementary information in free‐text narratives. Both groups provided similar levels of causality‐relevant information (p = .774). HCPs reported earlier than consumers (p < .05).

Conclusion

HCPs and consumers provided highly complementary unstructured information and comparable completeness for causality assessment. Collaboration between these groups should be encouraged, particularly in public health and clinical emergencies like epidemics and in situations of high uncertainty.

What is already known about this subject

• Consumer reporting to pharmacovigilance has risen, especially during mass immunization programmes.

• Consumers and healthcare professionals emphasize different details in spontaneous reports, often of comparable clinical relevance.

• Qualitative analysis of free‐text and studies using unequivocally matched duplicates in mass immunization contexts have been scarce.

What this study adds

• We thematically analysed free‐text from 127 unequivocally matched duplicate reports after immunization.

• Narratives were highly complementary (85% pairs); healthcare professionals added dose details, tests and causality reflections; consumers added supplements, life consequences and subjective explanations.

• A new completeness score showed similar causality‐relevant information across reporters; healthcare professionals reported earlier.

1. INTRODUCTION

Healthcare professionals (HCPs) in Norway can report suspected adverse drug reactions (ADRs) to the Norwegian Medical Products Agency (NOMA) since 1971, ¹ and specific electronic ADR forms for consumers were introduced in 2010. ² These electronic forms are based on the ICHE2B ADR reporting form ³ and include mandatory fields, such as the suspected medicinal product and the suspected ADR, alongside optional structured and unstructured fields. Since April 2020, they may include the consumers' national identity number or other directly personally identifiable information. ¹ , ⁴ The consumer reporting form includes a unique unstructured field for information about dietary supplements, as well as a specific questions about weight and height, which are not present in the HCP version of the form. ⁵ Otherwise, the consumer and HCP forms are largely identical.

In the last 15 years, an increasing proportion of consumers has contributed to the pharmacovigilance (PV) registries both in Norway and internationally. ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ The prevalence of suspected ADR reporting by consumers gained increased visibility during the COVID‐19 vaccination campaign, which in Norway started in 2021 and resulted in the administration of over 13 million confirmed COVID‐19 vaccine doses. ¹² NOMA received over 60 000 suspected ADR reports on COVID‐19 vaccines as the suspected medicinal product in the Norwegian Adverse Drug Reaction Registry (NorADRR), of which over 50 000 came from consumers. ¹² , ¹³ , ¹⁴ The increase in individual case safety reports (ICSRs) sent by consumers can be a valuable resource in signal detection as consumers who experience the adverse events (AEs) may identify ADRs that their HCP is unaware of and report new and different suspected ADRs to HCPs. ¹⁵ , ¹⁶ , ¹⁷

Previous research suggests that the type of information provided by different sources might not always overlap, with consumers more often describing symptoms, while HCPs tend to focus on medical history and test results. ¹⁸ Paired analyses of suspected duplicate ICSRs, in which the same event was likely reported both by consumers and by HCPs, indicate that different reporter groups disclose information of comparable clinical relevance. ¹⁹ Existing comparisons between consumers and HCPs identified duplicate ICSRs without definite confirmation of consumer identity and were outside of acute health or emergency situations. ¹⁸ , ¹⁹ , ²⁰

The comparative contributions of consumers and HCPs to PV registries during health emergencies, such as the COVID‐19 vaccination campaign, remain unclear. Certain information reported by consumers might be difficult to use in PV analyses, for instance, because interpreting and coding consumer ICSRs with standard medical dictionaries can be challenging for PV officers. ²¹ Although consumers' ICSRs supported prompt signal detection in the COVID‐19 pandemic, causality evaluation was time‐consuming when symptoms were subjective, difficult to standardize and not easily defined within formal clinical criteria. ²² , ²³ Despite the general agreement on the importance of case narratives for causality assessment in PV databases, ²⁴ , ²⁵ there is no systematic knowledge about the comparative quality and content of case narratives in patient reports. Understanding how consumers and HCPs report suspected ADRs can clarify their respective contributions and reveal how their information can complement each other to support causality assessment. This research question is best addressed by comparing how consumers and HCPs report the same event, and the implemented system in NorADRR in 2020 gives this unique opportunity.

This study aimed at comparing how Norwegian HCPs and consumers reported the same suspected ADRs following immunization with vaccines against COVID‐19 during the COVID‐19 pandemic. Specifically, we aimed at comparing the extent to which HCPs and consumers reported information relevant for assessing the causal relationship between the AE and the suspected COVID‐19 vaccine, with a particular focus on the type of unstructured information provided by each group.

2. METHODS

2.1. Study design

A retrospective study was performed using duplicate ICSRs, where the same AE following immunization with COVID‐19 vaccines was reported to NorADRR from both consumers and HCPs.

2.2. Data sources

In 2020, Norway updated its legislation whereby the Adverse Drug Reaction Registry Regulation was adopted. ¹ , ⁴ This required reporters to enter the consumer's national identity number or another direct personal identifier when filling out electronic ADR forms. Given that the Norwegian national identity numbers are unique for each individual and cover date of birth and gender, ²⁶ its use enabled linking information about the same occurrence submitted by different sources to NorADRR. ¹⁴ This system ensured confirmation of reports referring to the same individual and AE, so‐called duplicate ICSRs, and permits confirmation of individual identity in de‐duplication work. In addition, the use of the national identity number in multiple registries allowed verification of vaccine administration and date in the Norwegian Immunization Registry (SYSVAK). ²⁷

2.3. ICSR selection

The study included the initial version of valid electronic ICSRs ³ , ²⁸ reported to NorADRR between 2021‐01‐01 and 2023‐12‐31 with a COVID‐19 vaccine as the suspected medicinal product. Staff at NOMA confirmed duplicate pairs of ICSRs by national identity numbers and the administration date of the vaccine in SYSVAK. At least one ICSR in the duplicate pair was classified as ‘serious’ in NorADRR, even if the reporter may not explicitly have marked it as ‘serious’ according to the EMA criteria. ²⁹

ICSR duplicates identified by NOMA were accessed in NorADRR. Predefined details in each ICSR were tabulated (see Appendix 1) with two or more persons present to avoid human data collection errors. When an ICSR had two or more HCP reporters for the same consumer, the consumer ICSR was included twice – consumer compared to HCP 1 and consumer compared to HCP 2.

Exclusion criteria involved ICSRs in the following circumstances: There were fatalities or data protection concerns, the consumer was under 18 years, the consumer was found to be an HCP, ICSRs were in paper format, ICSRs from the pharmaceutical industry or the initial ICSR from one reporter was not available due to data protection requirements. ¹⁴

2.4. Analysis of unstructured text

Thematic analysis was carried out to identify the information contained in the unstructured text extracted from ICSRs. Information in each ICSR was coded independently by at least two researchers. In total, three researchers were involved in the coding process (T.E.D., J.S. and E.R.) and any disagreements were resolved by discussion to reach concurrence. All three coders piloted 20 ICSRs collectively for initial calibration. Codes were generated inductively during the whole process. Codes were subsequently grouped together into themes, each code corresponding to a category of information. A matrix was then used to annotate the categories of information contained in each ICSR. The level of complementarity between duplicate ICSR pairs was measured by number of categories of information present in only one ICSR of either duplicate, divided by the total number of categories of information present overall in both duplicate ICSRs. Inter‐rater agreement, quantified as percent agreement before consensus, was calculated as the number of identical coder decisions divided by the total number of coding decisions.

2.5. Quantification of ICSR completeness for the purpose of causality assessment

We developed a novel scoring system to quantify the ICSR completeness for the purpose of causality assessment. Our selection of information elements relevant for causality assessment was guided by prior research identifying which data support signal identification and verification ³⁰ and by a review of the inputs required by existing causality‐assessment tools systems. ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ Literature‐derived elements were subsequently refined via consensus‐based discussions among authors with practical and research experience in causality assessment. The scoring system involved categorizing presence or absence of information in either structured or unstructured fields of the ICSRs (Table 1). For each information element, the score combined its importance for causality assessment (importance weight) and the precision of the information provided (precision grade). We assigned importance weights to information elements based on their prevalence across existing causality assessment tools. ³⁹ ‘Time to onset’, ‘alternative or exclusory causes for the AE’ and ‘rechallenge information’ appear in all methods and therefore carry more weight in the composite score compared with other elements such as ‘chronology of AEs’ or ‘medical history’. Within each element, we then graded the available information on a scale reflecting precision and completeness, with higher grades indicating more specific, detailed or quantifiable information. Rules for point allocation are reported in Table 1.

TABLE 1.

Completeness scoring rubric for COVID‐19 vaccine ICSRs: elements, point values, criteria, evidence sources and examples.

Information elements	Points	Criteria for point scores	Source	Example
Name of the suspected medicinal product(s)	1	Ambiguous/misspelled/incorrect. Product identifiable only with external verification.	S, U	‘Acyclovir Pfizer’
	2	Precise name or brand documented in the ICSR.	S, U	‘Comirnaty’
Information about the AE(s)	1	Symptom/diagnosis only mentioned in structured field, no descriptors.	S, C	‘Fever’
	2	Symptom/diagnosis plus descriptors on diagnosis/symptoms/correlated events/localization.	S, C	‘38.9 °C fever with chills and myalgias’
Time to onset	0	Insufficient to compute time to onset (one or both dates missing or too vague).	S, C	‘Same year’
	2	Approximate timing only (non‐numeric/approximate timeframe).	S, C	‘Same month’ ‘few days later’
	4	Exact timing available (numeric interval in hours/days/weeks or both dates fully specified).	S, C	‘Two days after vaccination’, ‘Onset day 2’
Chronology of AE(s)	0	No information on progression/sequence/duration.	C	‘Joint pain’
	2	Unstructured text describes progression/sequence/duration.	C	‘Increasing pain, lasted three days’
Rechallenge information	N/A	Only first dose of COVID‐19 vaccine given at time of reporting.	S/U	‐
	0	Previous vaccination undocumented, outcome of the rechallenge not reported.	S/C	‘Got dose n. 2’
	1	Previous vaccination documented, outcome of the rechallenge not reported.	S/C	‘Vaccinated twice with Comirnaty’
	4	Previous vaccination documented, outcome of the rechallenge reported.	S/C	‘Headache after both doses of Comirnaty’
Outcome of the AE(s)	0	Outcome provided in structured field only.	S/C	‘Recovered’
	0.5	Outcome provided in structured field with additional information on the outcome in the narrative.	S/C	‘Resolved within 48 h without treatment’
Clinical observations	0	No information on clinical observations.	C	‐
	2	Clinical observation(s) documented.	C	‘BP 110/65 and normal pulse’
Test results	0	No test results reported.	C	‐
	2	Any laboratory/diagnostic test result reported (qualitative or quantitative).	C	‘CRP 36’ ‘Elevated liver function tests’
Concomitant medicine(s)	0	No information on concomitant medicines/supplements.	S/C	‐
	1	At least one concomitant medicine/supplement named. No dose/regimen or timing.	S/C	‘Metformin’
	2	Name(s) plus either dose/regimen or timing; or explicit statement of no concomitant medicines/supplements.	S/C	‘Metformin since 2018’
Medical history	0	No medical history reported.	C	‐
	2	Medical history or baseline health status prior to the COVID‐19 vaccination described.	C	‘Previously healthy’
Alternative cause(s) considered/ruled out	0	No information on alternative causes or exclusions.	C	‐
	4	Alternative cause or exclusion explicitly mentioned with supporting details (e.g., test/procedure and result or clear clinical rationale).	C	‘Normal CT excludes ischemic cause of chest pain’
Batch number	0	Batch number not provided.	S	‐
	0.5	Batch number provided.	S	‐
Total possible score	23 27	Maximum score when rechallenge N/A Maximum score with rechallenge

Points reflect relevance and precision of each element; higher scores indicate greater completeness. Items most informative for causality (time to onset, rechallenge and alternative causes) carry higher maxima. Maximum score: 23 when rechallenge is not applicable (only first dose at time of reporting); 27 otherwise.

AE = adverse event; N/A = not applicable; S = structured ICSR field(s); U = unstructured ICSR field; C = qualitative code from narrative.

For each ICSR, the final score was expressed as a percentage of the maximum possible score. The final percentage scores were graded to level of usefulness for causality assessment based on thresholds ‘well’ at >70%, ‘moderate’ at 40%–70% and ‘poor’ at <40%. The thresholds were chosen by qualitative sensitivity evaluations of candidate cut‐off points. For each candidate boundary, we randomly sampled at least five ICSRs within ±5 percentage points of the proposed boundary and compared cases immediately above versus below. The assignment of the boundaries focused on the presence of the key elements (‘time to onset’, ‘alternative or exclusionary cause for the AE’, ‘rechallenge information’) and the total amount of additional elements. We chose 70% and 40% as the thresholds because reports >70% typically included at least two key elements plus additional elements, whereas reports just below were generally missing one of the key elements or multiple additional elements; reports <40% were consistently deficient in both.

2.6. Calculation of time to reporting

Time to reporting was calculated for each ICSR as the difference in days between section C.1.4 in ICHE2B form ‘Date Report Was First Received from Source’ and the earliest date in section E.i.4 ‘Date of start of reaction/event’ or section H.1 ‘Case narrative’ in the ICHE2B form when both dates were present (see Appendix 2).

2.7. Statistical analysis

McNemar's test was applied to test for differences in nominal levels of information within duplicate pairs. The threshold for significance was set at p < .05 and two‐sided 95% exact (Clopper–Pearson) confidence intervals (CIs) were used. Descriptive statistics were used to examine the completeness characteristics of ICSR information. Wilcoxon signed ranks tests were performed for non‐normally distributed data. Confirmation of data normality was performed with histogram and Q‐Q plot visual inspections, Shapiro–Wilk tests and skewed distribution. The threshold for significance was set at p < .05 and CIs were two‐sided 95% for the Hodges–Lehmann median difference (related samples). Statistical analysis was performed using IBM SPSS Statistics 31 (SPSS, Chicago, IL, USA). ⁴⁰

2.8. Research ethics

This research was carried out in accordance with relevant guidelines and regulations. The regional research ethics committee south‐east Norway concluded that the project fell outside the scope of the Norwegian Health Research Act (reference number 589978, 21.02.2023). Data collection and storage were designed in consultation with the Norwegian Agency for Shared Services in Education and Research to ensure adherence to data protection and privacy regulations (reference number 914574, 15.08.2024). Because direct identifiers were not accessible, obtaining individual consent and issuing personal notices were impracticable; accordingly, an exemption under GDPR Article 14(5)(b) was invoked, and collective information about the study was provided via the DMP website.

3. RESULTS

3.1. Sample characteristics

At the start of data collection on 2024‐10‐01, NOMA had identified 178 unique cases of AE following immunization with COVID‐19 vaccines which were reported by both a consumer and at least one HCP. Of these, 14 cases were reported by two HCPs resulting in 192 pairs of duplicate ICSRs. Exclusion criteria were met in 65 separate occasions, meaning that 127 pairs of duplicate ICSRs met the inclusion criteria (Figure 1). Seven consumer ICSRs had two corresponding HCP ICSRs (HCP 1 and HCP 2), resulting in 120 unique consumer ICSRs.

FIGURE 1.

Flowchart showing exclusion criteria and duplicate ICSR pairs' selection. The health crisis timeframe involves ICSRs received between 2021‐01‐01 to 2023‐12–31.

The included ICSRs involved 120 consumers and 127 HCPs. Consumers consisted of 80 women and 40 men, and the majority were between 36 and 65 years old (Table 2). Fifty‐nine consumers (corresponding to 62 consumers in duplicate pairs) received their first dose of COVID‐19 vaccine. The 127 HCPs were classified with Norwegian protected professional titles as follows: physicians (105), nurses (14), physician and nurse (1), health secretaries (4), healthcare worker (1) and other unknown HCPs (2).

TABLE 2.

Consumer demographics and suspected COVID‐19 vaccines ¹ in the included ICSRs. Numbers indicate unique number of consumers. Parentheses indicate number of consumers included in two duplicate pairs.

Sex	Age group (years)	Comirnaty	Spikevax	Vaxzevria	Jcovden	Total
Women	18–35	10	6 (1)	2	0	18 (1)
	36–50	13 (1)	7	6 (1)	0	26 (2)
	51–65	14	6	6 (1)	0	26 (1)
	66–80	10	0	0	0	10
	Total	47 (1)	19 (1)	14 (2)	0	80 (4)
Men	18–35	5	5	0	0	10
	36–50	7 (2)	5	1	1	14 (2)
	51–65	12	0	1	0	13
	66–80	3 (1)	0	0	0	3 (1)
	Total	27 (3)	10	2	1	40 (3)

¹ Comirnaty composition was one of tozinameran, or tozinameran and famtozinameran, or tozinameran and riltozinameran, or tozinameran and raztozinameran. Spikevax composition was one of elasomeran, or elasomeran and imelasomeran, or elasomeran and davesomeran, or andavasomeran. Vaxzevria composition was: COVID‐19 vaccine (ChAdOx1‐S [recombinant]). Jcovden composition was: COVID‐19 vaccine (Ad26.COV2‐S [recombinant]).

3.2. Categories of information reported in free‐text narratives

The thematic analysis of unstructured text yielded five main themes, with a total of 19 categories of information, described and exemplified in Table 3. All themes and categories of information were present in ICSRs from both consumers and HCPs, albeit with different proportions. Initial percent agreement between coders before consensus was 84.6% across all category coding (N = 4826), ranging from 70.9% for the category ‘additional symptoms in the narrative beyond those captured in structured fields’ to 98.4% for the category ‘mention of similar AEs in other individuals’.

TABLE 3.

Themes and categories of information derived from the thematic analysis of ICSR case narratives. Examples are given for each category of information. AE = adverse event.

Main theme	Category of information	Description	Examples from consumers	Examples from HCPs
Grasping AE elements	More symptoms	Additional symptoms are presented in case narrative than provided in structured field(s)	‘Headache’, ‘Fatigue, muscle/joint pain’ and ‘Breathless’ in structured fields and in ‘swollen lymph nodes’ and ‘suspected swollen thyroid’ in case narrative	‘Headache’ in structured fields, and ‘Brain fog, concentration problems and forgets a lot’ in case narrative
	Description of the diagnosis	The diagnosis provided in the structured field is explained in more detail	‘Blood clot’ in structured field with the case narrative elaborations ‘vision became lethargic’ and ‘parts of the right side were very impaired, could nearly not walk and cannot write with right hand, and got some issues with speech, something I still notice now’	‘Mini‐stroke’ in the structured field with the case narrative elaboration ‘diplopia and right side and extremity paraesthesia’
	Chronology of AEs	Time progression, sequence of events, duration or temporality regarding AEs. Might include time to onset, but not necessarily so.	‘It started with itching and eventually it spread over the entire body. Rash that just got bigger and bigger minute by minute’	‘The evening after vaccination she got hives on both legs, stomach and scalp, and the next morning there was light numbness in the mouth and on the tongue – could not speak clearly, described a little swelling on the throat but denied having breathing or swallowing issues’
	Time to onset	The time taken until AE occurs after administration of the vaccine	‘Wavey feeling when walking on the right side of the head and acute dizziness came about after 25 minutes’	‘Started with headache circa 25 minutes after vaccination. Circa 40 minutes after vaccination the patient began slurring and could not find words’
	Localization	The specific location on the body the AE affects	‘Chills, severe pain in arm, neck, head, back and hand’	‘Developed pains and hydropsy in ankle joint. Later developed arthritis with swelling in left wrist”
	Subjective explanation	Details on how consumers experienced the adverse reactions	‘Suddenly get a stiff/sore jaw and I feel numbness. When I walk I sometimes have no energy in both thighs, they get numb prickling’	‘Feeling of wool in the head, blurred vision and dizziness’
Effects and management of AEs	Outcome	Statement on whether the AEs are still ongoing, resolving, resolved with sequelae or fully resolved	‘As of today the muscle aches in the legs, nausea, general bodily discomfort, brain fog and dyspnoea are in recovery and are mostly bothersome on “bad” days. The fatigue however, is still not any better’	‘This last week some light improvements. But after coming here today she feels fatigue. Gets very tired of walking outside, not reduced strength or power, but a feeling of exhaustion. Was difficult to get here to the 3rd floor. But she now manages one little thing per day’
	Life consequences or sequelae	The consequences in life the AEs have had or the sequelae the consumer has as a result of the vaccination	‘2–3 menstruations per month. 5–7 days each time, now spotting with only 1–2 days in between bleeds. Also difficult to go to the toilet’	‘Headache with debut circa 12 hours after vaccination with astra Zeneca. Intensity and localisation changes. Tormented daily. Has been on sick leave afterward some periods and at this time she is still on sick leave’
	Treatment	Description of how the AEs have been treated	‘During the last “outbreak” that allegedly was in the form of herpes meningitis (<date>), I received intravenous Valtrex for 4 days in the hospital. After that circa 8 months Valtrex treatment 1 g/day (ended start of <month>) I have not had any outbreaks’	‘After one week's duration the outbreak became so severe that she in addition developed herpes simplex meningitis and received antiviral treatment at XXX. No late sequelae after the infection’
Consumer profile	Medical history and/or clinical background	A description of the consumer's medical history, clinical status or health status before administration of a COVID‐19 vaccine	‘I have known asthma that is well regulated with flutiform (fluticasone and formoterol) and possibly ventolin (salbutamol), I have pollen allergy. From before I am healthy and very active, physically fit, and have been a cross‐country skiing athlete before. Do not smoke or drink’	‘Previously healthy more or less. No use of medicines’
	Concomitant medicines	Providing specific names of medicinal products used by the vaccinee concomitantly, or statement of no medicine use. The medicinal products are not considered as dietary supplements	‘I use Spirix (spironolactone) regularly’	‘Does not take medications’
	Dietary supplements	Providing information on dietary supplements used concomitantly to the vaccine administration, or absence of dietary supplement use	‘C‐vitamin, D‐vitamin, B‐12, Chrome, Folic acid’	‘Started over the counter iron supplement (“Lift iron and C‐vitamin”)’
Helping establish a diagnosis	Test results	Laboratory or clinical results or a description of test results	‘No findings in blood tests, CT or neurological examination’	‘CRP 36, leucocytes 7.1, Hb 10.6, d‐dimer 1.87, troponin‐T < 5. CT pulmonary embolism protocol without signs of PE’
	Clinical observations	A description of a visual or clinical check of the Consumer's health status	‘Goose bumps on thighs. Eventually tingling and prickling in hands and legs …. Other symptoms were crackling sounds when breathing, swelling over the entire body, especially stomach, venous congestion’	‘Symptoms in the form of chest pressure and dyspnoea, worsening during physical activity and when she lies down flat. No fever’
Contextual information and clinical reflection	Reflections on causality	Thoughts and reflections on possible causes of the AEs and the connection with the vaccine	‘The doctor could not establish whether it was pregnancy, caesarean section or the vaccine or a combination as the cause to the blood clot’	‘It is as such not certain that the infarction was a direct consequence of the vaccine, but she has no history that can explain any clinical infarction previously in life’
	Alternative or exclusory explanations to the AE	A description of other suspected reasons causing the reported AEs, or a description excluding possible causes of the AEs	‘Initially relatively healthy and good form (except sleeping disorder) with BP 110/65 and normal pulse. Allergy against nuts. Do not smoke or drink. Never had coronary issues or been given treatment before. No psychological conditions or anxiety problems. After first dose Pfizer/Comirnaty I felt health the first days‐ only pains in left arm. After ca 6 days chest pains in the middle of the chest and shortness of breath during lighter activity … Have to make it clear that this is incredibly real and it is in no way related to any kind of “anxiety” for vaccines etc’	‘In the course of investigation with CT coronary angiography to exclude coronary disease as cause of pain, which was normal’
	Vaccine dose	The vaccine dose number administered in the vaccination series against COVID‐19	‘Started to feel discomfort in the high 8 days after 1st dose of AztraZeneca vaccine was administered’	‘Thrombophlebitis left thigh 2 weeks after first dose of AZ vax’
	Rechallenge outcome	The outcome of reintroducing any COVID‐19 vaccine, whether positive or negative rechallenge	‘Strong headache and tired after first vaccine. This faded until vacation and did not notice any more before dose two was administered. After dose two the problems got bigger. Strong headache and tired/weak with subsequent sick leaves’	‘Has been afflicted by strong pressing frontal headache since dose 2 of Pfizer, as well as pronounced fatigue … In addition, had similar headache after dose 1 of Pfizer, but it passed within 14 days’
	Mention of similar AEs in others	Providing information about other people experiencing similar AEs	‘I request that this adverse effect is taken seriously. I know that this is not affecting just me as I have spoken with many others who experience big deviations’	‘Was examined by cardiologist as likely adverse effect of Moderna vaccine, he has had several cases with young, previously healthy who had developed tachycardia’

3.3. Category of information overlap between consumers and HCPs

We wished to analyse to what extent the categories of information provided as unstructured text by consumer and HCP overlapped. On a duplicate pair level, the unstructured text of 108 out of 127 (85%) duplicate ICSRs had >50% complementarity regarding the reported category of information (Figure 2). Twenty‐four duplicate pairs had 100% non‐overlapping information categories in the ICSR case narratives, and only three had ≤25%. In 13 of the 24 duplicate pairs with 100% ICSR case narrative complementarity, one of the two ICSRs had zero categories of information present. Other examples with 100% complementary information include the consumer providing information on localization, treatment, description of symptoms from the structured field, outcome, consequences in life or sequelae and dietary supplements taken, while the HCP provided information on test results, alternative or exclusory causes and reflections on causality (see Appendix 3). The case narrative duplicates with the most overlapping categories of information (76.9%) had 10 categories of information in common and three were complementary (Appendix 3). There were no duplicate ICSR pairs with more than 80% category of information overlap in the case narratives.

FIGURE 2.

Distribution of the duplicate pairs into five levels of complementarity of information, from a minimum level of 25% complementarity or less to a maximum level of 100% complementarity.

The median number of categories of information present in ICSR case narratives was six for consumers and five for HCPs, while the highest number was 14 for both consumers and HCPs. There was no significant difference in the number of categories of information given by consumers and HCPs in duplicate pairs of ICSRs (median paired difference, HCP to patient = 0.5 categories; CI = −0.5 to 1.5). Chronology and outcome of AEs were the most frequently provided categories of information by both reporter types (Figure 3). The categories of information provided more often by consumers than HCPs were dietary supplements taken (Δ consumers to HCPs = 0.39; CI = 0.33 to 0.39), life consequences or sequelae (Δ = 0.25; CI = 0.14 to 0.34) and subjective explanation of AEs (Δ = 0.17; CI = 0.08 to 0.24). HCPs provided more frequently than consumers information about vaccine dose (Δ = −0.24; CI = −0.33 to −0.12), clinical observations (Δ = −0.22; CI = −0.31 to −0.11), test results (Δ = −0.15; CI = −0.24 to −0.04) and reflections on causality (Δ = −0.14; CI = −0.20 to −0.05).

FIGURE 3.

Categories of information present in ICSR case narratives, duplicate pair analysis. Blue denotes that the category of information was present only in the consumer case narrative, red denotes that the category of information was present only in the HCP case narrative, and grey denotes that both the consumer and HCP provided the category of information. The results for dietary supplements taken was 0% for HCPs only and 2% for both consumers and HCPs. For mention of similar events, it was 3% for consumers only, 2% for HCPs only and 0% for both consumers and HCPs.

3.4. Comparative analysis of information completeness for the purpose of causality assessment

Consumers and HCPs reported information at the same level of completeness in 51 out of 127 duplicate pairs (Table 4). In the remaining pairs, either the consumer (34 pairs) or the HCP (42 pairs) outperformed the other by one or two levels, with HCPs more frequently scoring higher. Notably, both reporters scored ‘well’ in only five duplicate ICSRs, while 46 pairs showed one reporter scoring ‘well’ and the other scoring ‘moderate’ or ‘poor’. Analysis of the minimum information for a valid ICSR showed that 115 consumers and 111 HCPs submitted the required information to the highest completeness level. Three consumers and three HCPs (six out of 254 ICSRs) documented vaccine names in an incomplete way. Moreover, 10 consumers and 15 HCPs documented AEs in an incomplete way. The level of completeness for inclusion of information elements other than minimal valid ICSR elements was lower by both consumers and HCPs. Our data suggest that the HCPs were slightly more inclined to fill out the optional information elements. However, there was no statistically significant difference in completeness score between consumers and HCPs (0.65 points; CI = −3.95 to 4.90).

TABLE 4.

Cross‐tabulation of ICSR usefulness ratings for causality assessment in matched consumer–HCP pairs (N = 127). Cells show counts (percent of all pairs). Grey cells indicate pairs with identical usefulness ratings (Well/Moderate/Poor).

Reporter	HCP
Consumer	Number of pairs by usefulness rating	Well (%)	Moderate (%)	Poor (%)	Total (%)
	Well (%)	5 (3.9)	13 (10.2)	3 (2.4)	21 (16.5)
	Moderate (%)	22 (17.3)	35 (27.6)	18 (14.2)	75 (59.1)
	Poor (%)	8 (6.3)	12 (9.4)	11 (8.7)	31 (24.4)
	Total (%)	35 (27.6)	60 (47.2)	32 (25.2)	127 (100.0)

The duplicate ICSRs were also compared by percentage difference in the numeric scores. Figure 4 shows that 33 ICSR duplicate pairs have a score difference <10% between reporters, and 75 ICSR duplicates have <20% score difference.

FIGURE 4.

Cross‐tabulation of ICSR usefulness ratings for causality assessment in matched consumer–HCP pairs (N = 127). Cells show counts (percent of all pairs). Grey cells indicate pairs with identical usefulness ratings (Well/Moderate/Poor).

3.5. Time to reporting

In 114 of 127 duplicate ICSR pairs, both the consumer and HCP provided the start date of the AEs, enabling paired comparison of reporting time. The median number of days for HCPs was 44 days (interquartile range: 104.5 days) whereas for consumers it was 76 days (interquartile range: 226 days), with detailed number of days on a paired level in Appendix 2. There was a statistically significant difference between the reporter groups; HCPs were more inclined to submit their ADR forms in fewer days than consumers (median paired difference, HCPs to consumers = −14.5 days; CI = −32.5 to 0.0).

4. DISCUSSION

4.1. Main findings

This unique study with 127 confirmed duplicate ICSR pairs found no statistically significant difference between HCPs and consumers in reported information that are relevant to assess the causal relationship between the suspected COVID‐19 vaccine and suspected ADR. Consumers provided more often than HCPs information on dietary supplements taken, life consequences or sequelae and subjective explanation of AEs, while HCPs provided more frequently than consumers information on vaccine dose, clinical observations, test results and reflections on causality. In 85% of 127 duplicate pairs, HCPs and consumers provided at least 50% complementary information in free‐text narratives. The time taken to report suspected ADRs was lower by HCPs compared with consumers.

4.2. Relevance and implications

To our knowledge, this is the first study comparing how consumers and HCPs reported the same event, where duplicate ICSR pairs were unequivocally confirmed as such using Norwegian identity numbers. A similar paired analysis, where the duplicate pairs were identified using patient's date of birth and gender, in addition to reported MedDRA term and suspected drug, was conducted previously by Rolfes et al. ¹⁹ The study compared the relevance of the clinical information reported and found no significant difference between the two reporting groups. Since the items included in their clinical assessment tool are also important for causality assessment, Rolfes et al. conclude that ICSRs from the two reporting groups are comparable when it comes to causality assessment. Here, we wanted to compare more directly to which extent ICSRs facilitate causality assessment, and we have accordingly developed a scoring system that weights type and completeness of the reported information based on their utilization in causality assessment tools. For instance, temporality, rechallenge and concomitant causes are given the highest maximum points because they are present in common to all the existing causality assessment tools, to the authors' knowledge. ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² , ⁵³

A very notable feature of our dataset is that only a small percentage of the ‘well‐documented’ ICSRs met this standard for both reporters in a pair. Among the ICSRs where one reporter scored ‘well’, 46 out of 56 reporters in the duplicate pairs outperformed their counterpart's score category by at least one level. This may reflect the challenge of meeting the highest standard on our scale or the influence of contextual factors. In any case, it underscores a degree of complementarity in the reporter groups' contributions to causality assessment (Table 4).

Previous studies comparing the type of information reported in ICSRs by consumers and HCPs have indicated that the two reporting groups might complement each other. ¹⁸ , ⁵⁴ , ⁵⁵ , ⁵⁶ These observations, already described, are corroborated and reinforced by our study using a robust method, in particular by the paired quantitative analysis of unstructured text. Consumers and HCPs most often used unstructured text to report different categories of information, emphasizing different aspects of the same event. Consumers more often provided subjective explanations of AEs and life consequences or sequelae, as well as information on dietary supplements (the latter has to be expected because it is explicitly asked in the consumers' reporting forms). HCPs more often provided objective information, including the vaccine dose administered, clinical observations and test results. However, we did not find a difference in the reporting of medical history, which was reported more frequently by HCPs according to Rolfes et al. ⁵⁴ The difference might be due to the fact that we focus on unstructured information, while Rolfes et al. included attachments such as hospital discharge letters, which may include medical history.

We are not aware of previous studies comparing the way consumers and HCPs use unstructured text. We urge that the issue is timely and needs investigation, since case narratives are essential in PV analyses. ²⁴ , ²⁵ In our study population, we have found that during the COVID‐19 vaccine safety monitoring both consumers and HCPs used the unstructured text actively to report serious events. Narratives by both reporting groups were of comparable richness and contained substantial additional details, beyond those provided in structured fields, relevant for the purpose of causality assessment, such as descriptions of chronology, positive rechallenge, reflections on causality and alternative or exclusory explanations to the AE(s). Furthermore, unstructured text was used to elaborate and refine structured information, such as symptoms description and diagnosis. This is opposite to the previously described tendency to incomplete ICSRs, which scarcely support causality assessment. ²⁰ , ⁵⁷ The richness of the narratives in our study sample might be due to the seriousness of the reported events or to the public attention to PV during the COVID‐19 vaccination campaign or might reflect a selection bias due to the fact that the event was considered relevant enough to be reported by both sources. In any case, it shows that consumers can contribute with meaningful case narratives, useful to causality assessment and complementing the HCP narratives.

Timely reporting of safety concerns during health crises, such as the COVID‐19 pandemic, enables rapid regulatory decisions. For example, on 11 March 2021, Norway discontinued the AstraZeneca COVID‐19 vaccine based on timely safety reports and analyses, impacting the national immunization strategy. ¹³ , ⁵⁸ , ⁵⁹ In this study, HCPs took fewer days to report ICSRs than consumers, in line with previous results. ¹⁷ This might be due to the fact that HCPs have a reporting obligation to report new, serious, life‐threatening or fatal AEs. ⁴ , ⁶⁰ Consumers might also be unaware of reporting possibilities or may not feel they benefit from reporting suspected ADRs. ⁶⁰ , ⁶¹ , ⁶² , ⁶³ Delays in reporting is multifactorial and may depend on the reporter being able to recognize the possibly experienced AEs following immunization or other medicinal products, type and duration of suspected ADR, the reporter's availability to report suspected ADRs, the method of reporting and type of healthcare contact. Since this and previous studies demonstrate the relevance and quality of consumers' contributions, it is worth considering the extent to which reporting delays by consumers may hinder preparedness in health emergencies. Further research should investigate this issue.

4.3. Strengths and limitations

A strength in this study is that the consumer identity in duplicate ICSRs were verified using the Norwegian identity numbers regardless of reporter type, allowing to compare ICSR reports that with certainty described the same events. Additionally, the qualitative and quantitative components in study design complement each other to identify what kind of information is provided by consumers and HCPs in ICSRs and whether the information provided is useful for causality assessment.

However, this study has limitations. The generalizability of the findings to other demographic populations, medicinal product types or health emergencies is uncertain due to our convenience sampling. Our sample included only serious cases, which were considered relevant enough to be reported by both HCPs and consumers independently This selection likely increases the absolute completeness of both report types compared with single‐reporter cases and may attenuate consumer–HCP differences due to information sharing between reporters. Consequently, generalizability to settings in which only a consumer or only an HCP submits a report is limited. Our quantitative scoring system is based on the presence, partial presence or absence of specific information elements and does not require subjective expert evaluations, such as assessing the clinical relevance of the information. While this approach enhances the reliability of the scale, it does not account for the fact that the impact of certain information elements on causality assessment depends on their clinical relevance. For example, for recurring events such as ‘menstruation disturbances’, specifying the exact date of onset in terms of hours or days may be neither feasible nor clinically relevant. Consequently, our scale might have been overly stringent in such cases, as it does not allow for achieving the maximum score for the information element ‘time to onset’. Nevertheless, since our analysis focused on paired comparisons between reporters, and not on the absolute value of the individual scores, this limitation is unlikely to have significantly influenced the final conclusions. Finally, quantitative scoring was performed by a single researcher using pre‐specified, rule‐based point allocation to (i) the codes derived from unstructured text and (ii) the structured information recorded during data collection, as indicated in Table 1. Although we did not quantify inter‐rater reliability for the final scores, the risk of rater‐dependent bias is mitigated by (i) independent double coding of the unstructured text with low percent disagreement before consensus; (ii) two‐person, cross‐checked collection of information from structured fields; and (iii) rule‐based point allocations that limit additional subjective judgement. Nevertheless, residual rater‐dependent bias cannot be excluded.

5. CONCLUSION

Our analysis shows that while ICSRs from consumers and HCPs have comparable completeness level for the purpose of causality assessment, they contain complementary details about the reported phenomenon. The level of complementarity emerging from both qualitative and quantitative analyses illustrates that both HCPs and consumers should continue to report suspected ADRs. Collaboration between these groups should be encouraged in all contexts of PV, particularly in health emergencies like epidemics and in situations of high uncertainty. During the COVID‐19 vaccine safety monitoring, reports often lacked enough details to support a proper causality assessment of the case. ²³ We propose that a collaborative ADR reporting, where consumers and HCPs identify relevant information through an open discussion, might both improve the ICSR completeness and at the same time decrease the variability of the event descriptions. Our proposal is in line with previous arguments encouraging stakeholder collaboration and an active inclusion of lay users in PV, in order to enhance preparedness to detect unexpected events that might otherwise be unnoticed. ⁶⁴ , ⁶⁵

AUTHOR CONTRIBUTIONS

Tommy Emil Dzus helped finalize the study aim, conducted the preliminary data collection, preliminary and final data analysis in consultation with the other authors and drafted the manuscript. Johaina Sarrage conducted the preliminary data collection and data analysis. Hilde Samdal suggested the aim of the study and was responsible for identification of ICSRs (material preparation). François Montastruc made important intellectual contributions, consulted in data analysis and helped finalize the study aim. Elena Rocca defined the aim of the study, conceived the study design, was responsible for consultation with the Norwegian Agency for Shared Services in Education and Research to ensure adherence to data protection and privacy regulations, organized data collection and contributed to the final data analysis and drafting of the manuscript. All authors gave feedback and proposed changes to the first manuscript draft. All authors read and approved the final version of the manuscript.

CONFLICT OF INTEREST STATEMENT

All authors have no relevant conflicts of interest to declare.

Appendix 1. Fields collected from ICSRs in ICH‐E2B (R3) format

Information type collected	Field in ICHE2B adverse event form	Format collected	Example of how information was recorded
Date report was first received from source	C.1.4	Presence	Day/month/year
Qualification	C.2.r.4	Exact information	Physician
Date of birth	D.2.1	Presence	66–80 years
Sex	D.5	Exact information	Female
Reaction/event as reported by the primary source in native language	E.i.1.1a	Exact information	Urticaria
Date of start of reaction/event	E.i.4	Presence	Day/month/year
Time until reporting ICSR	E.i.4 and C.1.4	Calculation based on exact information	39 days
Date of end of reaction/event	E.i.5	Presence	Day/month/year
Outcome of reaction/event at the time of last observation	E.i.7	Presence	Present
Characterization of drug role	G.k.1	Exact information	Suspect
Medicinal product name as reported by the primary source	G.k.2.2	Exact information	Comirnaty
Strength (number)	G.k.2.3.r.3a	Presence	Present
Strength (unit)	G.k.2.3.r.3b	Presence	Present
Date and time of start of drug	G.k.4.r.4	Presence	Day/month/year/time
Time to onset	G.k.4.r.4 and E.i.4	Calculation based on exact information	7 days
Date and time of last administration	G.k.4.r.5	Presence	Present
Batch/lot number	G.k.4.r.7	Presence	Present
Dosage text	G.k.4.r.8	Exact information	Dose 2
Pharmaceutical dose form (free text)	G.k.4.r.9.1	Presence	Present
Route of administration (free text)	G.k.4.r.10.1	Presence	Present
Case narrative	H.1	Exact information	Patient had itching and bumps. She took contact with her physician. Blood tests showed no abnormalities.
Reporter's comments	H.2	Exact information	Sender qualification: physician

Appendix 2. Time taken to submit electronic ADR forms after the start date of adverse reactions by reporter category, a paired analysis. Cells indicating duplicate ICSRs with the same time range until submitting the ADRs are marked in grey.

Reporter		HCP
	Number of days	≤10	11 to 30	31 to 90	91 to 183	184 to 365	≥366	Total
Consumer	≤10	8	7	3	5	1	0	24
	11 to 30	5	3	3	6	0	0	17
	31 to 90	3	5	5	6	0	0	19
	91 to 183	5	1	2	9	0	2	19
	184 to 365	8	3	7	1	2	1	22
	≥366	2	1	4	3	2	1	13
	Total	31	20	24	30	5	4	114

Appendix 3. Narrative examples from consumers and HCPs in duplicate ICSRs by level of complementary information. Footnote references are given below the table

Case number	Level of complementary information based on categories of information (%)	Consumer narrative	HCP narrative
1	100	‘A lot of pain over stomach, behind ribcage on the right sideA. Blood thinner injectionsB for 6 months. Lowered activity, fatigueC. Still feel some pain in the same place with a lot of activity and stressD. Have stopped workingE. Got a new job as a X now from 1st February. I take Nyco plus MultivitaminsF’.	‘CT/MR liver confirmed portal vein thrombosisG. Investigation in the hospital has now been able to find any other causes to the vein thrombosis so farH. Possible side effect from Comirnaty has to be suspectedI. The patient was hospitalised at X’.
2	100	‘I was very dizzy the first 2–3 weeks after vaccinationJ, the dizziness has come back in periods and has been so incapacitatingK that I am on sick leaveE. Am being examined by chiropractor and ear‐nose‐throat specialist. I have never before the vaccination suffered from dizzinessL’.	‘Ca 10 hours after vaccinationM: rapid onset of nausea, severe headache, joint pain in back and legsA. Now 10 days after vaccination still pain in body and nauseaD. Getting betterD’
3	23.1	‘Took Vaxzevria <date>N and got a strong reaction afterward (fever, muscle aches, fatigue, stomach aches, nausea, headache, physical weakness, sleeplessness) but all side effects passed after a few daysJ,M,O. Completely healthy until <date> when I got ComirnatyN. This time also a strong reaction with the same side effects as the previous timeP, but with vomiting, breathlessness and chest pains in addition. Was admitted to the emergency room and bed ward with chest pains 24–48 hours after the vaccine, but the pains dissipated, and CT of lungs showed no blood clotG. Most of the adverse effects passed after a couple of days like with the first doseD,N. The fatigue, muscle aches in legs, feeling unwell/nausea, general bodily discomfort, brain fog and breathlessness did not passD,K. Was bedbound the first month with these adverse effects and could not go up a staircase without getting heavy breath and all the symptoms were amplified so I had to rest on the top of a staircase before I could continue walkingE. As of today the muscle aches in the legs, feeling unwell/nausea, general bodily discomfort and breathlessness are on the mend and are mostly pronounced on “bad” daysD,K. The fatigue on the other hand is not improvingD and I am on sick leave 100%, as I have been since the second doseE,N. I function very poorly in everyday life’.	‘Got Asta Zenecavaccine <date>N got a strong reaction for 2 days with fever and muscle aches, a bit worn out for a couple of days more, but after that feeling wellJ. Got the Phizer vaccine <date>N Same night worsened general state of health, feverish feeling and chills, muscle aches, loose stools and vomitingM,O,P. From the afternoon of <date> chest pains radiating to the backA. No heavy breathing, no cough, no stomach painsQ. Was admitted to the hospital to rule out PE, pulmonary embolism was disprovedG and patient was discharged. Got strong headache that gradually passedJ. But after, severe fatigue. It is now 7 weeks since the vaccine and she is still exhaustedD. Was healthy beforeL. Discuss psychosocial factors and there is nothing in her life that can explain exhaustionL. The first 6 weeks she was so exhausted that just to walk around in the house was burdensome. Had to gather herself to go to the toilet, a lot of bedrest. Now the past week a bit on the mend. But after having taken herself here today she feels fatigue. Gets very tired from walking outside, not reduced strength, but a feeling of exhaustionK. It was difficult to walk up here to the 3rd level. But manages to do a little bit every day, for example meeting someone. Has to rest the remainder of the day. Cannot even think about exercise or workE’.
4	25	‘Started ca 3 weeks after dose 1M,N, woke up to go to work <date> but struggled to get up on my feet. Was sent to doctor and was referred to the hospital. Took plenty of tests without them finding anything wrongH. Had very low D vitamin and low folic acidG. Got tablets that I still take todayB. Got gradually better after ca 2–3 months with sick leaveD,E. Took 2nd dose of vaccineN and got worse again after ca 1 weekM. All adverse effects came about the same time in arms and legsA,P. Now today arms are completely fine but I am still struggling with my legsD,K. Cannot do my work normally. Also get tired very quickly and energy levels are very reducedO. As of today there is stiffness in the legs, pains in legs, shaking in legs during times I take the stairs or sometimes normally, also struggling with sleep and must use arms to get up if I for example do a squatE. Take vitamin D and folic acidF’.	‘He got 1st dose of corona vaccineN, Pfizer, mid <month>. <Date> he got a pronounced weakness in both arms and legsM and the day after he was hospitalised at X. Also complained about prickling in face, arms and legsA. Clinical observationQ, MR caput and spinal fluid examination was normalG,H. Was confirmed low D‐vitamin and folateG. Got supplementsB,F and normalisation of blood tests at health check 6 weeks later. Still muscle weakness and visible twitching in musculatureD. Referred to neurological department again. Normal neurophysiological examination. Gradually recovering from symptomsD. He took the 2nd doseN, Pfizer, <date> whereafter he became weaker mainly in the legs, he feels he has little energy and he has felt prickling in the face a few times for a duration of ca 5 minutesK,P. He has got the twitching back in the muscles but less often than in <month‐month>D. The patient was hospitalised at X’.

Note: A = localization, B = treatment, C = description of symptoms, D = outcome, E = consequences in life, F = dietary supplements taken, G = test results, H = alternative or exclusory causes, I = reflections on causality, J = chronology, K = subjective explanation, L = medical history, M = time to onset, N = vaccine dose information, O = more symptoms than structured field, P = rechallenge outcome, Q = clinical observations, <date> = a date, anonymized for GDPR reasons, <month> = a month, anonymized for GDPR reasons.

DATA AVAILABILITY STATEMENT

All data supporting the results of this article are in the NorADRR, available for research purposes after approval from the NOMA. The collected and analysed raw research data are available from the corresponding author upon request.