Unraveling controversies over civic honesty measurement: An extended field replication in China

Significance

Civic honesty is universally important since it promotes trust, respect, accountability, and cohesion. Previous research using a single measure of civic honesty may have overlooked the cultural variations in civic honesty–related behaviors. This study adopted dual measures of civic honesty, incorporating a theoretical lens of individualism/collectivism to examine how cultures can shape behaviors consistent with civic honesty. Our findings reveal that the variations in collectivistic/individualistic values can explain the differences in behaviors across cultures, which has significant implications for understanding cultural differences in civic honesty. Our study helps resolve the controversy surrounding an influential field experiment, refines civic honesty measurement, and advances the understanding of this critical concept.

Abstract

Cohn et al. (2019) conducted a wallet drop experiment in 40 countries to measure “civic honesty around the globe,” which has received worldwide attention but also sparked controversies over using the email response rate as the sole metric of civic honesty. Relying on the lone measurement may overlook cultural differences in behaviors that demonstrate civic honesty. To investigate this issue, we conducted an extended replication study in China, utilizing email response and wallet recovery to assess civic honesty. We found a significantly higher level of civic honesty in China, as measured by the wallet recovery rate, than reported in the original study, while email response rates remained similar. To resolve the divergent results, we introduce a cultural dimension, individualism versus collectivism, to study civic honesty across diverse cultures. We hypothesize that cultural differences in individualism and collectivism could influence how individuals prioritize actions when handling a lost wallet, such as contacting the wallet owner or safeguarding the wallet. In reanalyzing Cohn et al.’s data, we found that email response rates were inversely related to collectivism indices at the country level. However, our replication study in China demonstrated that the likelihood of wallet recovery was positively correlated with collectivism indicators at the provincial level. Consequently, relying solely on email response rates to gauge civic honesty in cross-country comparisons may neglect the vital individualism versus collectivism dimension. Our study not only helps reconcile the controversy surrounding Cohn et al.'s influential field experiment but also furnishes a fresh cultural perspective to evaluate civic honesty.

Civic honesty is essential for a well-functioning society, as it reflects individuals’ reliability and ethical conduct in fulfilling their civic obligations as citizens (1–3). Although civic honesty is a universally acknowledged concept, its behavioral manifestations can differ considerably across societies and cultures (4–6).

To evaluate global levels of civic honesty, Cohn et al. (2019) conducted an international experiment where researchers turned in “lost” wallets at urban institutions in 40 countries. These wallets contained business cards with an email address as the only mode to contact the “owners.” Half of the wallets were randomly assigned to contain money, while the other half did not. Cohn’s study revealed that wallets with money had a significantly higher email response rate than those without money (7). The email response rate was the only criterion employed to rank civic honesty levels across countries.

Cohn et al.’s study garnered considerable global attention, yet it also ignited several controversies, particularly concerning about China’s lowest ranking and the validity of relying on the email response rate as a universal benchmark of civic honesty for all 40 countries, without accounting for their unique cultures and social norms (8–15). The behavioral manifestations of civic honesty are multifaceted and culturally specific, with varying interpretations and corresponding codes of conduct across different cultures (4–6). Previous research has criticized using the email response rate alone, as it failed to consider the degree of email accessibility (16). However, we contend that a more fundamental issue with this measure is that it overlooked the significant role of culture in shaping behaviors that demonstrate civic honesty. In contrast, a culturally sensitive and multidimensional approach, with supplementary measures to augment the email response rate, should be adopted in cross-cultural comparisons of civic honesty (16).

Across social norms, keeping others’ wallets is deemed dishonest or unethical (17). However, ethical ways to handle a lost wallet may vary, including contacting the owner via different ways of outreach (e.g., sending an email or phone call) or safekeeping the lost property to enable the owner’s retrieval of it, both of which reflect behavioral manifestations of civic honesty (18–21). The way honest individuals choose to handle a lost wallet, either contacting the owner proactively or safekeeping the wallet passively, can be influenced by their cultural inclinations toward individualism vs. collectivism. Specifically, individualistic culture tends to prioritize self-directed actions toward a goal, while collectivistic culture favors actions that consider others and reflect a sense of group belonging (22, 23). In an institutional setting such as in Cohn’s study, emailing the wallet owner represents an agentic action typical of an individualistic culture, while safekeeping is more behaviorally inhibited and socially restrained, typical of collectivistic culture (24, 25). However, societies have varying degrees of individualism and collectivism, so individuals in a particular society may make decisions and manifest civic honesty through behaviors with a mixture of individualistic and collectivistic responses (26, 27). To our best knowledge, no study has examined the relationship between individualism/collectivism and civic honesty, but such knowledge may help resolve the controversies surrounding cross-cultural comparisons of civic honesty through behavioral manifestations, as demonstrated by Cohn et al.’s study. As (28) suggested, the relative emphasis on individualism vs. collectivism is potentially one of “the most important dimensions of cultural difference in social behavior” (28, 29).

To address this knowledge gap and overcome cultural insensitivity associated with using a single measure of civic honesty by behavioral manifestations, we replicated Cohn et al.’s study in China with significant extensions. Replications are crucial for advancing science, especially in the social sciences (30–32). National academies of sciences across countries emphasize replication as an “essential part of science” (33, 34). Cohn et al. (2019) also called for replication studies of their research in their eLetter (7). Our study extended the replication by comparing civic honesty levels in China using two measures of honesty-related behaviors, email response and wallet recovery, and including an examination of the relationship between collectivism/individualism and behavioral manifestations of civic honesty. To broaden the latter analysis, we also utilized Cohn et al.’s original data to examine the relationship between collectivism/individualism and email response rates across countries. Our finding indicated that using the email response rate alone can significantly underestimate the level of behavioral manifestations of civic honesty in China as compared to using a more culturally sensitive measure. Furthermore, we found a significant relationship between collectivism/individualism and alternative measurements of behavioral manifestations of civic honesty both within China and among different countries.

Results

Wallets were dropped at 500 institutions in 10 cities in China in 2019 with randomization of “money” vs. “no money” conditions (n = 250 for each condition) and randomly assigned with undercover observers (n = 250 for each group). Due to uncontrollable reasons (e.g., office closure or non-cooperation with the wallet-dropping process), four sites were removed from the experiment, which reduced the total sample size to 496 (248 for each “money” vs. “no money” condition). In 250 sites, undercover research assistants were set up to clandestinely observe the wallet-dropping process and recipients’ initial handling of the wallets. The descriptive statistics of the recipients across the 496 sites are presented in SI Appendix, Tables S1 and S2 and section 5, which shows balanced samples between the money condition and the undercover observation. Moreover, we also compared the statistics of the sites reported by the foreign research assistants who dropped the wallets and those reported by the undercover observers (SI Appendix, Table S3 and section 5). Some characteristics, e.g., understanding the situation and whether a security guard was on the scene, were significantly different (P < 0.05), which may suggest that foreign research assistants were not able to accurately record some variables given time limitations and the challenges of detecting subtle norms, e.g., differentiating the subjects between security guards and other employees.

Table 1 presents the email response rate, wallet recovery rate, and complete wallet contents recovery rate by “money” and “no money” conditions. Just as in Cohn et al.’s study, the email response rate was significantly higher in the “money” group than in the “no money” group (32.7% vs. 22.2%, respectively, P = 0.009). However, there was no significant difference regarding the wallet recovery rate between both conditions (78.6% in the money group vs. 77.0% in the no money group, P = 0.666). Moreover, the complete wallet contents recovery rate of the “money” group was significantly lower than that of the “no money” group (65.7% vs. 75.0%, respectively, P = 0.024), which was the opposite of the results using the email response as the outcome. For a more detailed accounting of the missing items, please see SI Appendix, Table S4 and Fig. S1; note that there was no significant difference in the missing items between “money” and “no money” wallets (P > 0.05). Remarkably, the average email response rate (27.4%) was much lower than the average wallet recovery rate (77.8%) and the average complete wallet contents recovery rate (70.4%) in the overall sample. The divergent results indicated that different conclusions can be drawn regarding civic honesty in China if using alternative measurements.

Table 1.

Email response, wallet recovery, and complete wallet contents recovery outcomes by money conditions (N = 496)^*

	No Money		Money		Total Sample
	%	SD	%	SD	%	SD	P-value
Email response	22.2, 7.0†	0.416, 0.256†	32.7, 21.5†	0.470, 0.412†	27.4, 14.3†	0.447, 0.350†	0.009, 0.000†
Wallet recovery	77.0	0.422	78.6	0.411	77.8	0.416	0.666
Complete wallet contents recovery	75.0	0.434	65.7	0.476	70.4	0.457	0.024

^*Four sites’ wallet recovery results were not obtained due to various reasons (e.g., office closure).

^†Response rates from the China study in ref. 7.

Notes: The last column presents P values for the null hypothesis of perfect randomization ( ${\chi }^2$ tests).

The results of the subjective attitudes toward the two measures of civic honesty are presented in Fig. 1. Fig. 1A shows the results from the survey of the employees in the replication study about their perceptions as to the relevance of different behaviors to civic honesty in China. Among the 347 employees who answered the question about whether failing to contact the owner was relevant to civic honesty, only 39.2% answered partially or very relevant. On the other hand, among the 372 employees who answered the question of whether retaining one or all items in the wallet was relevant to civic honesty, 80.9% answered partially or very relevant to civic honesty. Fig. 1B presents the results from a nationally representative online sample in China. A total of 2,420 respondents completed the online survey, while 2,310 were included in the analyses after the attention check. Of note, 84.6% of the respondents thought that retaining all or partial items in the wallet was dishonest, but 61.5% of them thought that failing to contact the owner was dishonest.

Fig. 1.

Attitudes toward alternative measurement of civic honesty.

Therefore, the summary statistics from the field experiment and the survey results from the employees and the general public all indicated that in China, safekeeping wallets by themselves conforms more with the social norms for civic honesty than the failure to contact the owner by email.

Table 2 presents the results of the linear regression analysis using the data collected by foreign research assistants. The probability of an email response increased by 10.21 percentage points (P = 0.011) in the “money” condition relative to the “no money” condition by controlling only the city and institutional fixed effects (model 1). With controlling the recipient’s characteristics and environmental variables as in ref. 7, the coefficient increased to 12.14 (P = 0.005) (model 2), which is close to the coefficients in ref. 7 (10.79, P < 0.001 for the 40-country sample; 15.01, P < 0.001 for China sample). The results suggest that the probability of an email response was higher in the “money” group even when controlling recipient’s characteristics and environmental factors, which is in accordance with Cohn et al.’s results (7).

Table 2.

Regression results of email response, wallet recovery, and complete wallet contents recovery from foreign research assistants’ perspectives^††

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)
	Email (N = 496)	Email (N = 496)	Email (N = 496)	Wallet recovery (N = 496)	Wallet recovery (N = 496)	Complete wallet contents recovery (N = 496)	Complete wallet contents recovery (N = 496)	Complete wallet contents recovery (N = 496)
Money	10.214*	12.142**	12.167**	1.399	2.162	2.046	−9.588*	−10.066*	−10.282*
	(4.011)	(4.325)	(4.335)	(3.621)	(3.817)	(3.812)	(4.032)	(4.287)	(4.259)
Male		−10.750*	−15.770**		−8.632*	−3.934		−10.433*	−2.509
		(4.434)	(5.790)		(4.134)	(6.033)		(4.633)	(6.286)
Age ≥ 40		−8.945	−8.054		−1.461	−1.736		3.317	1.723
		(5.280)	(6.400)		(5.371)	(7.169)		(5.782)	(7.209)
Computer		−3.757	−4.524		4.242	9.237		5.023	10.905
		(5.456)	(7.190)		(5.052)	(7.270)		(5.553)	(7.379)
Coworkers		−2.755	1.672		−2.021	−3.016		−4.326	−3.651
		(5.653)	(6.953)		(5.047)	(6.490)		(5.707)	(7.021)
Other bystanders		1.637	2.921		−4.584	−8.315		−1.635	−10.444
		(4.811)	(6.516)		(4.346)	(6.167)		(4.966)	(6.199)
% of rice paddies			0.074			0.456***			0.389**
			(0.131)			(0.116)			(0.133)
Controls:
City	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Institution	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
N	496	434	434	496	434	434	496	434	434
R 2	0.035	0.065	0.070	0.075	0.097	0.105	0.061	0.081	0.100
Adjusted R2	0.007	0.022	0.015	0.048	0.055	0.052	0.034	0.039	0.047

Notes: The OLS estimates are presented with robust SE in parentheses. The dependent variables “Email”, “Wallet Recovery”, and “Complete Wallet Contents Recovery” were assigned the value of 100 if a wallet was reported or recovered and 0 otherwise. “Money” is a binary variable for the treatment. In columns (2), (5), and (8), binary control variables were included to account for individual and situational factors, such as the recipient’s age (above 40 y), gender (male), and the presence of a computer, coworkers, and other bystanders. In columns (3), (6), and (9), a control variable “% of rice paddies” represents the percentage of cultivated land devoted to rice paddies in each province in 1996 (as explained by ref. 65). A complete set of interactions between the treatment and demeaned covariate of individual and institutional factors were included (69). Finally, all models include city and institution fixed effects. Significance levels: ^∗P < 0.05, ^∗∗P < 0.01, and ^∗∗∗P <0.001.

^††The regression results from undercover observers’ perspectives were not presented, because of the small sample size (n= 250), the outcomes were not significant.

Nevertheless, the money and no-money conditions were no longer significant (P > 0.05), and the coefficients were much smaller than that of the email response rates, if wallet recovery was treated as the outcome (1.40 for model 4 and 2.16 for model 5). In addition, if complete wallet contents recovery was treated as the outcome, the coefficients turned significantly negative (−9.59 in model 7, P = 0.018; −10.07 in model 8, P = 0.019), which suggests that the probability of complete recovery was significantly reduced if there was money in the wallet.

For models 3, 6, and 9 in Table 2, the percentage of rice paddies was added as a proxy of collectivism in the province where the city is located. The coefficient of the collectivism proxies was not significant when the email response rate was treated as the outcome, but was significant when wallet recovery was the outcome (0.456, P < 0.001). Similar results were found in complete wallet contents recovery. We also conducted a robustness check using a self-reported collectivism index in 10 cities as the measure of collectivism, and the findings were similar to those using the percentage of rice paddies as the measure of collectivism (see SI Appendix, section 6 for the methods and SI Appendix, Table S5 for the results). These results suggest that the collectivism at the provincial level was significantly related to the more traditional measure of civic honesty, that is, wallet recovery, but is not significantly related to the email response rate, which further confirmed the distinction between the two measurements of civic honesty in China.

We also asked the employees in the replication study whether they were aware of Cohn et al.’s study, which may have changed their behaviors in responding to the lost wallet. Among the 431 employees interviewed, 12 said that they knew of the Cohn et al. study (money group: 3; no-money group: 9). We conducted sensitivity analyses by excluding these samples, and the results remain qualitatively the same. Moreover, undercover observers (n = 250) reported that only 31.6% of the institutions had just one employee who participated in the wallet handling, while employees in the revisiting stage (n = 431) reported that only 19.0% of the institutions had only one employee to handle the lost wallet. These observations provide evidence of potential collective decision-making styles within the institutions as a driver of the civic honesty in the experiment.

Fig. 2 presents the relationship between the total email response rate and the collectivism index across the 39 countries (One country was not included for lacking collectivism index.) studied in ref. 7. There was a significant negative relationship between the total email response rate and the collectivism index of the country (Pearson’s correlation coefficient is −0.780, P < 0.01). Country-level collectivism index was still a significant predictor of the total email response rate even having accounted for the country-level internet penetration rate (SI Appendix, section 7). A robustness check was conducted to check the email response rates under the condition of money and no money separately (SI Appendix, section 8) and the results still hold. This evidence contributes to the literature by connecting one measure of civic honesty with collectivism at the country level.

Fig. 2.

Correlation between the email response rate in ref. 7 and the collectivism index across countries.

Therefore, through the statistical analyses in the field experiment in China and the reanalysis of the original data from Cohen et al., we found a connection between civic honesty measurement and collectivism across regions or countries.

Discussion

As the only attempt to replicate 7 in the field so far, this study successfully reproduced that study’s main finding: The email response rate was significantly higher in the “money” condition than in the “no money” condition. Nevertheless, this study also expanded our understanding of civic honesty in theoretical and empirical ways.

First, our study contributed to the field of civic honesty research by revealing the significant limitations of using a single, potentially culturally biased index to measure the level of civic honesty in a society. We found that relying solely on the email response rate as a measure of civic honesty could underestimate its actual level in China. Our results there using the wallet recovery rate were consistent with previous wallet dropping experiments (37). However, our use of additional measures suggests that exclusive use of the email response rate in Cohn’s study may have resulted in China’s lowest ranking and the resulting controversy. To ensure scientific validity and cultural sensitivity, it is essential to adopt multiple and appropriate measures of civic honesty, such as the wallet recovery rate, particularly in cross-cultural comparisons.

Second, this study introduces different components in the theoretical framework of the cultural underpinnings of civic honesty. Our cross-country comparison revealed a negative correlation between the email response rate and the country’s collectivism index, while our China study demonstrated a positive relationship between regional collectivism and wallet recovery. Collectivistic cultures, such as China, emphasize reservedness, cautiousness, and self-restraint over independence, sociability, and assertiveness (38–40). Group functioning requires behavioral constraint, obedience, and submission, which is why shy and inhibited behaviors are valued and encouraged (41, 42). Confucian philosophy, which underlies China’s collectivistic culture, considers inhibition and self-restraint to be signs of accomplishment, mastery, and maturity, with shyness, reticence, and inhibition viewed as demonstrating good behavior and understanding (43, 44). Engaging in autonomous behavior, on the other hand, is deemed selfish and socially unacceptable (41).

Consequently, in the lost-wallet paradigm, collectivistic cultures may express honesty through greater attention to wallet safekeeping, while individualistic cultures may prioritize establishing social interaction and approaching the wallet owners as desirable. Interestingly, the money condition increases the email response rate but not the complete wallet contents recovery rate. The email rate and wallet recovery rate may measure two distinct forms of civic honesty. Contacting the owner represents an agentic action typical of an individualistic culture (24, 25). Our findings suggested that the underlying collectivism/individualism of a culture may shape individuals’ beliefs about social norms, which drives distinct behaviors related to the civic honesty across cultures (45–47). This view highlights the diversity of human experience and promotes tolerance and acceptance of different cultural practices and beliefs. It is situated within the framework of cultural relativism, which suggests that people’s behaviors should be understood in the context of their own cultures (48–51).

To our knowledge, this research is the only attempt to replicate 7 in the field so far. It is especially worth noting that this replication was completed before the outbreak of the COVID-19 pandemic in 2020, which dramatically altered the conditions in the participating institutions, rendering any further replication attempts virtually impossible (52–54). Major public emergencies have been shown to significantly impact individuals’ perceptions of social norms (55, 56). In China, for instance, staff in institutions are now cautious about accepting items from strangers, particularly foreigners. Therefore, this study represents the only viable opportunity to replicate Cohn et al.’s study in China.

Although we have successfully replicated Cohn et al.’s field experiments in China and contributed valuable insights in the study of civic honesty, this study has several limitations. First, since the sample sizes in our study and Cohn et al.’s study were both relatively small, and the sampling was not nationally representative, we will not overstate the generalizability of the findings to the entire country. Second, the replication results obtained here cannot be generalized to countries other than China without further investigation. Third, respondents interviewed in the follow-up visit may have exhibited certain biases toward certain questions regarding the relevancy of the email response to the civic honesty if they did not contact the owner. Finally, while we selected institutions based on the criteria listed in Cohn et al., the specific institutions we chose may not have perfectly matched those chosen in Cohn et al., which was not disclosed publicly.

Despite its limitations, this study still serves as an important effort to both validate and extend Cohn et al.’s study on a critical and sensitive subject that has significant implications for national reputations. More importantly, our research opens a lens onto the role of cultural collectivism/individualism in understanding civic honesty, a key factor in maintaining an orderly and harmonious society. Further research and theoretical advancements are necessary to gain a more comprehensive understanding of how civic honesty should be measured across cultures.

Materials and Methods

Study 1: Extended Replication of Cohn et al. (2019).

Setting.

Cities and urban institutions.

We conducted the experiment in ten cities in China. Eight of them (Beijing, Shanghai, Guangzhou, Shenzhen, Tianjin, Chengdu, Hangzhou, and Xi’an) were the same as the cities in 7. Additionally, we added Harbin and Nanjing to increase the geographical representativeness of the replication in Northeast China and the Yangtze River area (Fig. 3).

Fig. 3.

The map of selected cities in the replication study in China.

We selected 500 sites across five types of institutions near the city center, which included 21.6% banks, 19.6% cultural attractions, 19% post offices or express delivery stations, 19.6% hotels, and 20.2% public offices (see SI Appendix, section 1 for sample size calculation). The reason for selecting express delivery stations was due to the decreasing presence of post offices near the city center, express delivery stations being the replacement. To replicate the Cohn et al. study, we selected appropriate institutions that fit the field experiment, e.g., they needed to be on the ground level and close to the street, so someone could return the “lost” wallet on the street to the staff in these nearby locations. SI Appendix, Table S6 lists the detailed breakdown of sample sizes by types of urban institutions in each city. The design of the field study followed the protocol of intentional deception in social sciences to avoid possible biases in the responses. However, the debriefing procedure was strictly implemented during the follow-up site visit as designed in the protocol. We have also received informed consent from the interviewees in the follow-up surveys. The study was approved by the Institutional Review Board at Zhejiang University in China.

Wallet replication and drop-off.

The dropped-off wallet fully replicates the design of Cohn et al.’s experiment (7), which contained the same personal items: three identical business cards, a grocery list, and a key (SI Appendix, Fig. S2). The business cards displayed the supposed owner’s name, email address, and job title, which were all in Chinese. We used the same owners’ names as in Cohn et al.: LI Qiang, CHANG Wei, and WANG Lei. Transparent business card cases ensured that recipients could inspect the wallet’s contents without having to open it. The business cards and shopping list served to identify the supposed owner as a local resident, signaling that it would be relatively easy to contact the owner and return the wallet. The key was to indicate the urgency of contacting the owner.

Since Cohn et al.’s (2019) main findings were about the significant difference in the email response rate between “money” and “no money” conditions, we randomly selected half of the wallets to have 52 RMB (Chinese currency) in the wallet, while half of them did not have money (7). The amount of money was based on the original amount (49 RMB) in Cohn et al. but adjusted for inflation from 2015 to 2019 (57). The 52 RMB consisted of one 50 RMB note and two one RMB notes.

Similar to the original study, foreign research assistants pretended to be strangers who picked up the wallet on the street and dropped it off to the employees in the institutions. They asked the employees to handle the wallets properly and then swiftly left the scene without leaving any contact information. Detailed protocols can be found in SI Appendix, section 2.

In the replication study, we added two components to the study design as follows (see SI Appendix, section 3 to compare the design of Cohn et al. and the replication study).

Undercover observers—1st extension to Cohn et al. (2019).

Cohn et al. relied on the foreign research assistants’ retrospective survey after the drop-off to collect information about the recipients’ sociodemographics and the environmental factors, such as whether there was a security camera or any other employee on the scene (7). The replication study also asked the foreign research assistants to complete the survey as in Cohn et al.

However, we identified three major limitations of Cohn et al.’s approach in our pre-experimental pilot studies: First, after foreign research assistants left the institutions, they were unable to observe how the recipient(s) handled the wallets, which created a significant information gap in understanding the relationship between the recipients’ behaviors and civic honesty. Second, the foreign research assistants were not necessarily able to differentiate between security guards, employees, or customers, who may, however, have been identifiable by local people, since locals are familiar with the subtle cultural cues conveyed through body language, clothing, or conversation. Finally, the foreign research assistants found that they were under pressure to complete the drop-off procedure and communicate with the recipient, while having little time to pay attention to all the other factors to be surveyed, e.g., how many bystanders were on the scene.

To overcome these limitations, we added undercover observers who were Chinese research assistants to the experiment, behaving as bystanders in these institutions to scan the institutional setting and observe the whole process from the drop-off to processing the wallet. However, the addition of the observers may have influenced the recipients’ decisions on how to handle the lost wallet, i.e., the audience effect (58, 59). Therefore, we randomized the institutions into two groups in each city: In the first group, the “environmental scan group,” observers only scanned the environment and left the institution before the foreign research assistants stepped in. In the second group, the “recording group,” observers recorded the complete drop-off process and stayed until the initial handling of the lost wallet was completed, e.g., the recipient put the wallet into the drawer.

To ensure the accuracy of the observations, each group consisted of two observers. One undercover observer used a hidden camera to record, while another undercover observer completed the survey, which was validated by viewing the video recordings retrospectively. See SI Appendix, Table S2 for the breakdown of sample sizes based on money conditions and the recording of the initial wallet handling status. Direct observation in public spaces was considered by the Institutional Review Board as exempt, with minimal risk to subjects, and using hidden cameras to record behaviors in public spaces is a common practice in behavioral studies (60–63).

Revisiting the institution—2nd extension of Cohn et al. (2019).

Seven d after the wallet drop-off, another group of Chinese research assistants revisited the institutions, claimed to be the supposed owners’ affiliates, and tried to recover the wallets if possible. The reason to pick 7 d as the cutoff point for revisiting is that 100% of the emails sent to the “owners” in Cohn et al. happened within 7 d in China. Regardless of whether the wallets were recovered or not, these research assistants, or “revisitors,” eventually disclosed their real identities and the nature of this scientific experiment in a short debriefing, which was required for human subject protection but not implemented in Cohn et al (64, 65). Afterward, the research assistants received informed consent from one employee who had assisted in the wallet recovery process and was asked questions about handling the lost item in the institution and their opinions about the concept of civic honesty. Two questions were asked about what was perceived as civic honesty:

• 1)“If someone gets the lost property and does not contact the owner, waiting for the owner to return to find it, do you think it is relevant to personal integrity?”
• 2)“If someone takes this lost property or part of it as his own, do you think it is relevant to personal integrity?”

These debriefing protocols and survey questions were tested and validated in the pilot studies prior to the experiments.

Since the employees may have had biases to answer these questions based on their own behaviors (e.g., failing to contact the owners), we conducted an online survey using a nationally representative survey in China (n = 2,420) to measure the general public’s attitudes toward civic honesty if failing to contact the owner and keeping the lost wallet in the wallet drop study environment (see SI Appendix, section 4 for details about the online survey).

Complementary measurements of civic honesty.

As in Cohn et al., we measured whether the recipients emailed the owner about the lost wallet as one measurement of civic honesty. Whether the wallet was eventually recovered serves as a complementary measurement of civic honesty, since this is the end outcome of the lost wallet processing. Moreover, whether all items were retrieved when the wallet was recovered, i.e., complete wallet recovery with no items missing, served as the third measurement of civic honesty. Although retaining possession of money may be more related to civic honesty than keeping other nonmoney items, we did not differentiate various missing items in this study but will analyze these specific items in a separate study. We hypothesized that a complete wallet contents recovery indicates a higher moral standard in civic honesty than a wallet recovery with some items missing, whether monetary or nonmonetary. Future research will examine the differences in the lost items.

Given these extensions to Cohn et al.’s original study, three binary indicators were created to measure civic honesty as the outcomes (whether there was an email response, whether the wallet was recovered, and whether the wallet was recovered without losing any items, i.e., complete wallet contents recovery). The main explanatory variable was the money condition of the wallet (1 = money; 0 = no money).

To understand the employees’ perspective on whether contacting the owner reflects civic honesty, we created two additional variables: whether failing to contact the owner of the lost wallet was relevant to civic honesty and whether retaining one or all items of the lost wallet was relevant to civic honesty. Both variables were measured with a seven-point Likert scale (0: not at all to 6: very relevant) to indicate the relevance of these behaviors to civic honesty. A binary indicator was created if the answers were five or six (partially or very relevant to civic honesty) when presenting the results.

Measurement of collectivism culture.

Talhelm et al. (2014) found that in China, rice growing requires a more interdependent culture, while wheat growing makes cultures more independent. The rice vs. wheat agriculture was a major explanatory factor of psychology and social behaviors in China, even after controlling for economic development and climate (66).

We used the percentage of rice paddies in the province to measure the collectivism culture in China as in refs. 66 and 35, which is an innovative measurement of collectivism across regions in China. Similarly, we used the same data sources to calculate the percentage of rice paddy lands among all cultivated areas in the province where the city is located (67).

Measurement of recipient characteristics and environmental factors.

As in Cohn et al., the recipient characteristics and environmental factors were recorded by foreign research assistants (7). Following Cohn et al.’s definition, the recipient’s characteristics include gender, age (40 y or older or not), whether the communication was in English, whether the situation was understood (Seven-point Likert scales from “not at all” to “fully understood”), and the busyness of the recipient (Seven-point Likert scales from “not at all” to “very busy”). The environmental factors included computer availability (whether there was a computer on the recipient’s desk), security camera availability (whether there was a security camera), and the presence of bystanders who might observe the drop-off (dummy indicators created for the condition of having coworkers, security guards, and other bystanders).

To provide more information about the wallet handling process, we created two additional variables regarding how many employees processed the lost wallet: First, undercover observers measured how many employees participated in the initial wallet handling after the foreign research assistant left; second, revisitors surveyed the employee to estimate how many employees participated in the wallet handling throughout the process. These two variables indicated the number of employees involved in the decision-making regarding the institution’s lost wallet.

Statistical analyses.

For the three outcomes of civic honesty, i.e., email response, wallet recovery status, and complete wallet contents recovery status, we present the binary outcomes, prevalence rates, and SD. For all independent variables related to the recipient characteristics and environmental factors, we calculated the percentage and SD for binary variables or mean and SD for continuous variables. For a two-group comparison (i.e., money vs. no money), we conducted χ²-tests for binary variables or Kruskal–Wallis H tests for ordinal variables, such as the level of understanding of the situation. To compare the civic honesty outcomes that were measured with binary variables, we conducted χ² -tests for a two-group comparison (e.g., money vs. no money).

Following Cohn et al., we used a linear probability model with ordinary least squares (OLS) regression and robust SE to examine the relationship between email response, wallet recovery, and complete wallet contents recovery and the money vs. no money conditions. We included two sets of specifications. First, we examined the effect of the money condition while controlling only the city and institutional fixed effects. Second, we adopted the full model with the percentage of rice paddies, recipients’ characteristics, and environmental factors. Freedman (68, 69) argued that applying OLS on estimating the treatment effect in randomized experiments may “lead to worsened asymptotic precision, invalid measurements of precision, and small-sample bias.” Lin (2013) suggested that adding a full set of treatment–covariance interactions can address Freedman’s concerns (36). Therefore, we applied the method introduced by ref. 36 to improve the model estimates from ref. 7, while controlling the recipient’s gender, age (40 y or older or not), computer availability, the presence of coworkers, and the presence of other bystanders. All study data are included in the paper and/or the attached SI Appendix. For all statistical analyses, we considered P values of less than 0.05 to be statistically significant. We analyzed all data using STATA, version 16 (70).

Study 2. Cross-country examination of civic honesty and collectivism.

We merged two datasets to examine the relationship between country-specific civic honesty and country-level collectivism. The measure of country-level civic honesty was estimated with the country-specific email response rates from ref. 7. The total email response rate was defined as the total number of emails received divided by the total number of experiments conducted in that country (money and no money condition together). The measurement of country-level collectivism was from Hofstede et al. (71), which is extremely widely cited. Hofstede divided national cultures into multiple dimensions, which include individualism as one dimension. He estimated the score of collectivism/individualism in 76 countries and argued that “individualism stands for a society in which the ties between individuals are loose… Collectivism stands for a society in which people...are integrated into strong, cohesive in-groups, which, throughout [their] lifetime, continue to protect them in exchange for unquestioning loyalty” (71). To be consistent with our analytical framework, we estimated the country-specific collectivism index by subtracting the individualism index from 100. Then, we merged the two datasets, email response rates, and collectivism index by countries. A scatter-plot figure was drawn, and the trend line was plotted.

Supplementary Material

Appendix 01 (PDF)

Dataset S01 (CSV)

Dataset S02 (CSV)

Data, Materials, and Software Availability

Data files, with excel csv format “Replication experiment and others_data (csv file).csv”; “Online national representative survey_data (csv file).csv”, are attached in the Supporting Information, which are also available in the public repository with link as: https://github.com/Science-replication-group/Unraveling_Controversies_over_Civic_Honesty_Measurement (72).