CLEAR	COMP.	VALID	FEASIBLE	SAMPLE STATEMENTS FROM EXPERT
Framework	Y	Y			Typically, a researcher will evaluate data quality/availability and develop a research design appropriate for the data. Recording a diagnosis code for "diabetes" in an EHR during a visit in which the clinician orders a test for diabetes is not necessarily an error, it might be a local policy that all diabetes tests ordered get coded with that diagnosis. I want to know whether my cohort has the right data available for a study, during the study period. It is one question for me, not three. These recommendations…are where this comes to life.
Constructs	0/3		0/3
Cells	4/9	1/3	3/9
Recs.	8/9			8/9
Framework	N	N			I think missing from the framework is actually the frame—when in the research process are we supposed to use this? It seems to be aimed at the analysis of a data set—after the data-collection process has been specified. [T]he concept missing for me is my data-quality workflow, as a quality assessor or researcher. [Regarding] the "pathway" of data from physical event to recording in the dataset. These three operationalizations don't cover all of them, so I presume you are making a choice based on some sort of tradeoff, having to do with ease of checking.
Constructs	0/3		0/3
Cells	1/9	0/3	6/9
Recs.	5/9			6/9
Framework	Y	N			Again it is contextual—fitness for purpose definition. But overall the logic of self-assessment and self-determination of what "sufficient" is makes sense. Progression on data over time reflects clinical course and will vary depending on a number of diagnostic, management and prognostic factors. So need constraints in framing the research question(s). I think the realist approach should be emphasized, i.e. the importance of context. The issue of "actors" is another important scope question as EHR-based research can be used for research about the care provider and interventions as well as impact on patients.
Constructs	3/3		2/3
Cells	6/9	0/3	6/9
Recs.	7/9			9/9
Framework	Y	N			I don't see anything to address the quality issue of, "Are the right patients included in the data?" Perhaps this is more of a research question…but it seems to cross into the data quality boundary when someone attempts to use the data for something that's not fit for purpose. [I]t feels as if [completeness] depends on the 'task at hand.' If the goal is to estimate an effect, then completeness requires that the estimate can be generated without bias due to confounding. [Using an] external reference is a good idea, but practically is quite difficult, both in terms of logistics and methodologically ensuring that the external reference should be comparable to the source population.
Constructs	0/3		2/3
Cells	9/9	3/3	9/9
Recs.	4/9			3/9
Framework	N	Y			Each construct seems like it should be followed by the term "for the task at hand." [The completeness across patients recommendation] may be difficult for larger datasets, composite variables, and deciding when to do this… where does this get represented? [For the current across patients recommendation,] this is clear—I'm not sure how feasible it is. [For the current across time recommendation,] without metadata for recording data I'm not sure how feasible this is.
Constructs	2/3		3/3
Cells	7/9	3/3	8/9
Recs.	8/9			7/9
Framework	Y	Y			I found the questions to be a nice way to frame the framework! I think the questions will help users understand the framework and the subsequently presented Recommendations. [Y]our framework fits into the larger picture of data quality, and into the frameworks created by others. These Guidelines beg the question of how these fit together. What about single-site versus multiple-site EHRs? What is the bigger picture? Does presentation of the final version of your Guidelines call for a short description of context?
Constructs	3/3		3/3
Cells	9/9	3/3	9/9
Recs.	9/9			9/9