Reply to Garcia et al.: Common mistakes in measuring frequency-dependent word characteristics

The concerns expressed by Garcia et al. (1) are misplaced due to a range of misconceptions about word usage frequency, word rank, and expert-constructed word lists such as LIWC (Linguist Inquiry and Word Count) (2). We provide a complete response in our paper's online appendices (3). Garcia et al. (1) suggest that the set of function words in the LIWC dataset (2) show a wide spectrum of average happiness with positive skew (figure 1A in ref. 1) when, according to their interpretation, these words should exhibit a Dirac δ function located at neutral (h_avg = 5 on a 1–9 scale). However, many words tagged as function words in the LIWC dataset readily elicit an emotional response in raters as exemplified by “greatest” (h_avg = 7.26), “best” (h_avg = 7.26), “negative” (h_avg = 2.42), and “worst” (h_avg = 2.10). In our study (3), basic function words that are expected to be neutral, such as “the” (h_avg = 4.98) and “to” (h_avg = 4.98), were appropriately scored as such. Moreover, no meaningful statement about biases can be made for sets of words chosen without frequency of use properly incorporated.

Garcia et al. (1) compare our work on English with a similar sized survey by Warriner et al. (4). Warriner et al. generated a merged list of 13,915 English words, the bulk of which are a list of lemmas taken from movie subtitles, a mismatch with the corpora we used in creating our English word list labMT (language assessment by Mechancical Turk). In figure 1B of ref. 1, Garcia et al. make a flawed comparison between the two word lists because the words behind each histogram are not the same. For shared words, the minor difference in median h_avg of 0.07—much less than the observed positivity bias—cannot be because of our use of cartoon faces (emoticons). The earlier Affective Norms for English Words (ANEW) study upon which we modeled our work (5) also uses cartoons and yet found a lower median for words shared with Warriner et al. (5.29 versus 5.44) (4). All three datasets agree well in more general statistical comparisons (4).

In attempting to say anything about a given quality of words as it relates to use frequency within a specific corpora, a complete census of words by frequency must be on hand, otherwise uncontrolled sampling issues arise. In Fig. 1A, we plot average happiness as a function of frequency of use for the word list Garcia et al. (1) created from Google Books. The scatter plot is clearly unsuitable for linear regression. We show an estimate of cumulative coverage at the bottom, which crashes soon after reaching 5,000 words.

Fig. 1.

(A) Scatterplot of h_avg as a function of word usage frequency for the English Google Books word list generated by Garcia et al. (1). Uncontrolled subsampling of lower frequency words yields a poor candidate for linear regression. The lower curve provides a coarse estimate of cumulative lexicon coverage as a function of usage frequency f using Zipf’s law f_r ∼ f₁r⁻¹ inverted as r ∼ f₁/f_r. The rapid drop off begins at around rank 5,000, the involved lexicon size for Google Books in labMT (2). (B) Scatterplot of h_avg as a function of rank r for the 5,000 words for Google Books contributing to labMT, the basis of our “jellyfish” plots (2). (C) The same data as in B but now plotted against the inverse of usage frequency. The approximate adherence to Zipf’s law f ∼ r⁻¹ means there is no substantive loss of information if regression is performed on the correct transformation of frequency. Linear regression fits for the first two scatterplots are h_avg ≃ 0.089 log₁₀ f + 4.85 and h_avg ≃ −3.04 × 10⁻⁵ r + 5.62 (as reported in ref. 3). Note difference in signs, and the far weaker trend for the statistically appropriate regression against rank in B. Pearson correlation coefficients: +0.105, −0.042, and −0.043 with P values 6.15 × 10⁻²⁶, 3.03 × 10⁻³, and 2.57 × 10⁻³. Spearman correlation coefficients: +0.201, −0.013, and −0.013 with P values 6.37 × 10⁻⁹², 0.350, and 0.350 (B and C must match). The Spearman analysis indicates that an assumption of a nonmonotonic relationship between h_avg and rank r is well supported.

Sampling issues aside, Garcia et al. (1) state that regression against frequency f is a better choice than using rank r because information is lost in moving from f to r. However, the general adherence of natural language to Zipf’s law, f ∼ r⁻¹, provides an immediate counterargument, even acknowledging the possibility of a scaling break (6). Fig. 1B shows how use rank is well suited for regression, and is the basis for the “jellyfish” plots we presented in our work (3). In Fig. 1C, we present how h_avg behaves as a function of 1/f, illustrating both the error in choosing log₁₀ f and that our results will be essentially unchanged if we regress against 1/f.