Table 3.

Concepts in reliability and validity assessment for potential product-level surveillance methods

Source	Method(s)	Reliability	Validity
Patents	Keyword search; content analysis; patent mapping	Consistency across searchers (agreement > 85%); prespecified search terms	Predictive (e.g., does a PREP come to market relying on a patent identified as important?)
Internal Tobacco Industry Documents	Keyword search; semiotics; content analysis	Trained, blinded coders; systematized prespecified categories; inter-coder agreement > 0.85	Convergent (e.g., does preliminary data in documents associate with patents or marketed products?)
Industry reports and communications	Content analysis	prespecified search terms	Face
Government Reports	Content analysis	prespecified search terms	Face
Correspondence with the company	Interviews; questionnaires	NA	Face;
Informants	Interviews	NA	Convergent (do different informants give similar information?)
Products introduced in international markets	Keyword search, interviews, questionnaires	prespecified search terms	Predictive (e.g., does the foreign product or similar technology eventually appear on the US market?)
Government databases	Frequency analysis	NA	Face
Trade publications	Content analysis	prespecified search terms
Magazine Advertising	Readership demographics Advertising volume Expenditures Placement information Content analysis Semiotics	Trained, blinded coders; systematized prespecified categories; inter-coder agreement > 0.85; sourcing from multiple companies	Predictive (e.g., do advertising expenditures and ad content lead to increases in sales or beliefs about product safety?)
Internet	Keyword search; content analysis; semiotics	Trained, blinded coders; systematized prespecified categories; inter-coder agreement > 0.85	Convergent (e.g., do blog postings about products correspond to sales and use data for those products?)
Sales and Use	Scanner data Tax receipts	NA	Convergent (e.g., do sales data and reported use correlate positively?)
Local Area Tracking	Synar inspections; retailer licensing	NA	Convergent (e.g., does product availability at retail correspond to sales and usage data?)
Population usage	Mall intercept; RDD; web survey; panels; focus groups	Test-retest; internal consistency;	Face; predictive; convergent; divergent