Stakeholder engagement increases transparency, satisfaction, and civic action

Significance

Many international organizations urge companies in the oil, gas, and mining sectors to engage with local communities, and they issue “best practices” guidelines for doing so. However, reliable information about the impact of stakeholder engagement on participant communities is limited. This paper rigorously examines, through a randomized controlled trial, whether communities benefit from these efforts.

Abstract

This study evaluates the effectiveness of a Stakeholder Engagement (SE) intervention in improving outcomes for communities affected by oil and gas extraction in Western Uganda. The study design is a randomized controlled trial where villages are randomly assigned to a treatment group (participating in SE) or a control group (not participating). Data are collected via household surveys at baseline and end line in 107 villages in the Albertine Graben. We find that SE improves transparency, civic activity, and satisfaction with issues that most concern the people under study. While satisfaction has improved, it is too early to ascertain whether these interventions improve long-term outcomes. These results are robust when controlling for spillover effects and other subregional fixed effects.

Many practitioners and policy makers encourage community involvement in natural resource management (1, 2). A popular model is Stakeholder Engagement (SE), where private companies, nonprofits, and local/regional/national governments involve local communities in cross-sector decision-making (3–5). The rationale for SE is that it can provide a voice to local communities, information to decision makers, improved cross-sector collaboration, and net benefits to all parties involved (6). However, there is little empirical evidence evaluating the efficacy of SE efforts (7–9). This study addresses some of those shortfalls through a rigorous experimental test of the benefits of SE in the extractives sector (oil and gas).

The complex governance of natural resources has been characterized as a “wicked problem” (10). Such problems exist where there is inherent uncertainty about the underlying policy problem, how it might affect various stakeholders, and the responsibilities of policy makers. Wicked problems are ubiquitous in natural resource governance but especially pronounced in the extractives sector: Communities are often peripheral or uninvolved in extractives decision-making (9, 11), have limited understanding of the technical complexities of extractives management, and typically have limited access to accurate information about extractives governance (12, 13). Many see SE as the key policy tool to overcome these hurdles (5).

But does SE actually benefit communities the way some of its proponents hope? Evaluating the effectiveness of SE presents a major research challenge: Companies select communities for SE strategically, so treatment is not random. Companies typically invest more resources for SE in those communities directly and acutely impacted by extractives development (9, 11), and these communities are likely systematically different from comparison communities.

To facilitate causal identification, we evaluate an SE program initiated and conducted by a Ugandan not-for-profit company, Maendeleo ya Jamii (MYJ).* MYJ has been engaging with and organizing diverse stakeholders in the Ugandan oil sector for a number of years.^† As we describe below, MYJ expanded their program into additional communities; we randomly invited half of these communities to participate in the collaborative processes organized and managed by MYJ.

Oil and Gas in the Albertine Graben

The Albertine Graben covers ∼25% (68,000 km²) of Uganda’s land, and is inhabited by 25% of Uganda’s population. Its southern portion is one of the most densely populated rural areas on the African continent (17). Petroleum exploration has taken place intermittently in the Albertine Graben for almost 100 y, but more-recent assessments have uncovered untapped reserves in the Lake Albert escarpment. Since 2002, 121 wells (39 exploration wells and 82 appraisal wells) have been drilled in the Albertine Graben. Of these, 106 wells have revealed 21 oil and gas discoveries, representing a drilling success rate of over 88% (18). Commercial quantities of oil were discovered in 2006, and preparations for the development of these discoveries are currently underway as exploration continues (18, 19). Prior work by MYJ revealed that local people have a broad variety of concerns about policy, transparency, consultation, and social services; however, the most prominent concern was the need for SE (17).

The study identified some districts affected by oil operations in western Uganda: Hoima, Buliisa, Moyo, Yumbe, Arua, Nebbi, Nwoya, and Ntoroko. Next, we limited the study to communities with populations between 200 and 2,000 according to the most recent census. We did so to ensure that we could obtain adequate community coverage within the household survey. We then eliminated all communities wherein MYJ had previously worked. Based on this sampling frame, we identified 391 potential communities for inclusion into the study.

A power analysis conducted with data from the baseline survey suggested that we needed over 100 communities for the study. Many of the 391 potential communities identified for the sampling frame lie geographically close to one another. Thus, we opted against pure random sampling to avoid spillover effects, and instead sampled 107 communities within these districts under the constraint that all sampled communities be at least 3 km away from any other sampled community. Fig. 1 shows all of the communities identified by the population constraint (hollow circles) as well as the final communities that were randomly selected under the distance constraint (crosses).^‡

Fig. 1.

Map of sampled communities. Created using ESRI software (37) using data from (38, 39) and authors’ data.

The Intervention

The intervention consists of 2 parts: information provision to local communities (available to both treatment and control villages) and participation in multistakeholder forums (MSFs; which were only accessible to treatment communities).

In both control and treatment villages (107 project villages total), MYJ staff delivered 2 hard copies of an information package during meetings convened by each Village Chairperson.^§ This information package is a compilation of questions and answers based on community and local government concerns about Uganda’s oil and gas activities in the Albertine Graben. These concerns have been elucidated over time by civil society organizations and central government agencies. The Village Chairperson was the custodian of the information packages. Copies of the information packet are found in SI Appendix.

At those same meetings, residents of the 52 treatment villages each selected 3 representatives to participate in one MSF. These representatives comprised the Village (LC1) Chairperson, and 2 others chosen by the community (one had to be female). Villagers were directed to select their 2 representatives based on 3 criteria: ability to communicate in English, confidence that they will effectively represent their respective villages and present concerns to the forum, and trust that they will provide feedback to the village after the engagement.

The treatment villages each experienced MYJ-facilitated SE for the first time. Three different MSFs were planned for the treatment, each having no more than 60 participants. The treatment villages were clustered by district, with Buliisa, Hoima, and Ntoroko as one group; Arua, Moyo, Nebbi, and Nwoya as a second group; and villages in Yumbe as the third. This 2-d engagement process involved the following: Day 1 was spent 1) interacting with community representatives from other oil-bearing villages to share their experiences; 2) interacting with the Association of Uganda Oil and Gas Service Providers and learning about private sector experiences in Uganda’s petroleum sector; and 3) interacting with the Ministry of Energy & Mineral Development as well as the Ministry of Lands, Housing & Urban Development to learn about the status of Uganda’s petroleum sector and discuss regional development plans. Day 2 was spent developing village action plans based on the priority concerns of the representatives’ respective communities, identifying what roles each community representative should play in executing these plans, and agreeing on a reasonable timeframe for completion.

At the end of the 2-d MSF, each team of village representatives left with a folder that contained a copy of all of the information presented to the forum, the information generated by the participants during group sessions (i.e., their respective village action plans), and the names and contact information of various government and private sector presenters. MYJ encouraged the representatives to share the information they had learned with their village members and to fulfill the commitments they made in their action plans.^¶

Communities in the treatment group are better prepared to mobilize, to represent their interests, and to demand accountability from the decision makers responsible for their concerns. Decision makers, in turn, become more aware of what communities’ concerns are and agree to specific commitments for each community. Also, if civic activity becomes more robust, then it becomes costlier for these decision makers to ignore their commitments to communities.

The Effects of SE

We report Difference-in-Differences (DID) estimates of treatment effects for the outcomes of interest agreed to by the researchers and MYJ. Materials and Methods reports further details on the DID estimation strategy. Program outcomes fall into 3 categories: Transparency, Civic Action, and Satisfaction. Table 1 shows mean estimates of each outcome at baseline and end line for the treatment and control groups, while Table 2 reports DID causal estimates of program impact. The 3 models reported in Table 2 represent estimations including/excluding village fixed effects and village-clustered SEs.

Table 1.

Summary statistics for outcome measures

	Control		Treatment
	Baseline	End line	Baseline	End line
Transparency Index*	−0.105 (1.01)	−0.061 (0.96)	−0.091 (1.02)	0.216 (0.99)
	[n = 1,107]	[n = 1,355]	[n = 1,083]	[n = 1,382]
Civic action: Household†	0.14 (0.73)	0.1 (0.54)	0.121 (0.65)	0.24 (0.74)
	[n = 1,518]	[n = 1,579]	[n = 1,499]	[n = 1,549]
Civic action: Community†	0.258 (0.99)	0.234 (0.81)	0.212 (0.81)	0.538 (1.08)
	[n = 1,314]	[n = 1,426]	[n = 1,304]	[n = 1,427]
Satisfaction Index‡	3.88 (1.85)	3.97 (2.37)	3.742 (1.78)	4.064 (2.41)
	[n = 1,581]	[n = 1,609]	[n = 1,566]	[n = 1,579]

Mean (SD), [number of observations].

^*Constructed using principal component analysis on a series of related survey questions.

^†The sum of different civic actions (out of 15 total) respondents reported at the relevant level.

^‡Extent to which a person’s satisfaction increased with issue areas she or he ranked as most important.

Table 2.

DID treatment effects

	Model 1	Model 2	Model 3
Transparency Index*	0.265*** (0.06)	0.237*** (0.05)	0.237*** (0.07)
Civic action: Household†	0.159*** (0.03)	0.156*** (0.03)	0.156*** (0.05)
Civic action: Community†	0.351*** (0.05)	0.349*** (0.05)	0.349*** (0.08)
Satisfaction Index‡	0.236** (0.11)	0.235** (0.11)	0.235** (0.11)
Village-fixed effects	No	Yes	Yes
Village-clustered errors	No	No	Yes

Estimate (SE), **P ≤ 0.05, ***P ≤ 0.01.

^*Constructed using principal component analysis on a series of related survey questions.

^†The sum of different civic actions (out of 15 total) respondents reported at the relevant level.

^‡Extent to which a person’s satisfaction increased with issue areas she or he ranked as most important.

Transparency.

Transparency refers to a culture of information sharing between citizens and oil sector decision makers, as well as an increased pursuit of information by citizens. We measure transparency by creating an index out of 6 questions asked in the household surveys (see SI Appendix for more details). These include survey questions on objective knowledge about oil and gas, as well as more-subjective questions about people’s perceptions of transparency. We perform principal components analysis to create a single weighted index of these different factors; the index has a mean of zero and SD of one.

Table 2 shows positive, significant effects on the transparency index. Our analysis shows that the treatment increases the transparency index from between 0.263 (model 1, P < 0.01) and 0.235 (models 2 and 3, P < 0.01) SD.

Civic Actions.

We asked each respondent about 2 types of civic actions related to the oil and gas sector: those taken by the household itself and those taken by their community. We also asked for further details about which of 15 different types of civic action were taken (attending further meetings about the oil sector, participating in CSO activities, etc.; SI Appendix). We code our civic action outcome measures as 1) the total number of activities each household reported engaging in itself and 2) the number of activities in which they reported their community engaging.

Table 2 shows positive, significant effects for both household and community civic actions. The DID estimates that the treatment increases the number of household actions to between 0.159 (model 1, P < 0.01) and 0.156 (models 2 and 3, P < 0.01). The DID estimates that the treatment increases the number of community actions from between 0.351 (model 1, P < 0.01) and 0.349 (models 2 and 3, P < 0.01).

Overall Satisfaction.

Our initial piloting and MYJ’s previous work suggested that land management, social service provision, and local economic development were the most salient issues for these communities. One difficulty of measuring change in these areas is that households may differ in assessments of their relative importance. MYJ’s prior work indicated that actors in the oil sector may address an area less important for some households or communities. Thus, we sought to measure change in satisfaction with those issue(s) each household deemed most important relative to others.

To gauge relative importance, we presented survey respondents with 3 note cards, each labeled with one of the issue areas described above. We asked respondents to allocate 10 stickers across the notecards, in proportion to how important they thought each issue was. After this task was completed, we placed 3 additional cards in front of them and asked them to place 0 to 10 stickers in proportion to how satisfied they were with the handling of each issue.

From these 2 tasks, we calculated a Satisfaction Index. We form the Satisfaction Index by using issue importance scores to weight up or down respondents’ reported satisfaction with each issue area. We then sum the weighted satisfaction scores across issue areas.^# This method is based on the idea that increased satisfaction with a policy area that the respondent cares little about is less meaningful than increased satisfaction with a policy area the respondent cares more about.

Table 2 shows positive, significant effect of the SE intervention on the index. The DID estimates that the treatment increases the satisfaction index from between 0.236 (model 1, P < 0.05) and 0.235 (models 2 and 3, P < 0.05) on a 10-point scale. Substantively, this means that the intervention increased respondents’ reported satisfaction with issue areas they prioritized.^||

Discussion

Although current policy guidelines tend to advocate SE in the oil extractives sector (20), most evidence, to date, only shows that such initiatives benefit firms (14, 15). There is less reliable evidence on the benefits these initiatives may or may not offer local communities (9). Our study represents a rigorous examination of that issue. We find that SE has significant effects over and above those of mere information dissemination. The SE intervention improved perceptions of transparency, increased civic activity, and improved people’s overall satisfaction with issues they deem most important.

We interpret these findings as conservative estimates of the impacts of SE efforts for 2 reasons. First, the measured impacts are still short-term and bear further examination as oil and gas development continues within the Albertine Graben. While we do have some anecdotal evidence that tangible service delivery is beginning to change in response to community actions, this bears watching over the coming years. Evidence suggests that, over a longer time scale, the provision of tangible benefits is an important predictor of local support for extractives development (21, 22). Second, there are reasons to believe that the short-term impacts of MSFs may be underestimated by our research design. MYJ’s prior experience indicates that MSFs appear most effective when they occur multiple times across a longer time period. MYJ also typically includes subcounty and district leaders in the MSFs they’ve conducted but did not do so for the purposes of this study, to help mitigate spillover effects. This exclusion may serve to weaken the estimated effects of the intervention.

Furthermore, some caution should be used when interpreting how effective SE is more generally, outside of the context of this study. One reason is that this research only speaks to a particular type of SE facilitated by a local organization that acts within the principles of direct, face-to-face engagement; gender equality in village representation; equality of opportunity to speak within MSFs; and support for local communities to develop action plans that should help them hold other decision makers accountable. The action plans also had a self-assessment template that encouraged communities to measure progress on implementation of actions over several weeks. Previous research suggests that principles and design choices like these are important determinants of the success of collaborative governance efforts (6, 22). SE efforts that do not follow these principles may not yield the same positive outcomes.

This SE program is in the extractives sector, but many areas of natural resources management face similarly “wicked” problems (10). For instance, land conservation also confronts problems like weak accountability mechanisms, fears of theft, frustration or inflated expectations about potential benefits, and a distrust of external influence (23–27). However, SE in extractives must also deal with some more-unique issues. Communities in the areas of extraction face unique obstacles, because they are usually not able to use the extracted resources and employment options tend to be limited. Benefits of natural resource extraction are usually seen in the national budget and do not trickle down to the affected communities. Also, community access to information and decision-making about extractives is often limited (9, 11–13). Meanwhile, extractives development may tend to draw more political attention (due to its potential value), more opportunities for corruption (28–30), and more fears of corporate malfeasance (12, 13). In short, extractives face extremely challenging problems, and the fact that we observe positive impacts from SE in this sector provides optimism that similar efforts might be successful in other sectors. However, more evidence is needed to establish this.

Materials and Methods

The research design is a prepost design with village-clustered random assignment to the treatment and control groups. Estimation of treatment effects is done using a DID estimator. This approach compares the change in an outcome between the baseline and end-line surveys among treatment village residents with the change in that outcome among control village residents over the same period. A positive effect indicates a greater increase between measures in the treatment group than in the control group.

Data.

We conducted baseline and end-line household surveys in all villages. Data were recorded through a smartphone survey application that was then uploaded to a secure server at the end of each day. We also added qualitative questions to better understand the motivations for certain responses.** All enumerators received training on both the qualitative and quantitative parts of the survey. The full study protocols were approved by the Florida State University Human Subjects Committee (SI Appendix). All subjects provided informed consent.

We used a quasi-random sampling technique (random walk) to survey ∼30 households per community. We used a quasi-random technique because no village lists exist from which we might randomly sample. Details are available in SI Appendix.

We conducted 30 household surveys in most communities. In sum, the survey team conducted 6,440 total (baseline/end line, treatment/control) household surveys, across 107 communities.^†† We emphasize that the sample is not a panel of respondents but separate random samples at end line and baseline. In addition, enumerator training emphasized an equal number of male and female respondents. The enumerator asked to speak to an adult knowledgeable about the household. In practice, the survey often drew interest from more than one household member. This was assessed daily using an enumerator activity log. The final sample comprises 48.8% men and 51.2% women.

The procedures to ensure a quasi-random sample of households within each community, the large quasi-random household sample sizes, and strong gender balance provide assurance that the information we obtained from the household sample is broadly representative of people within the region. We give details on the household survey and provide a copy in SI Appendix. Details about covariate balance (between treatment/control at baseline) and descriptive statistics are also reported in SI Appendix.

Estimation Strategy.

To account for the clustering of households within villages, we estimate DID treatment effects in a linear regression framework, where

$$y_{ijt}\hspace{0.17em}=\hspace{0.17em}\alpha \hspace{0.17em}+\hspace{0.17em}\beta Trea{t}_{ijt}\hspace{0.17em}+\hspace{0.17em}\gamma Endlin{e}_{ijt}\hspace{0.17em}+\hspace{0.17em}\tau Trea{t}_{ijt}\hspace{0.17em}\times \hspace{0.17em}Endlin{e}_{ijt}\hspace{0.17em}+\hspace{0.17em}{\epsilon }_{ijt}.$$

In this model, i indexes individual, j indexes villages, and t indexes time (baseline or end line). The estimate of the DID is the parameter $\tau$. We provide robust estimates of our SEs, clustered at the village level to account for cluster randomization (all households within a village are assigned to treatment or control group). This equation can, in principle, be modified to allow for a set of additional covariates to account for imbalances that remain after randomization. We turn to this in the robustness section.

Robustness Checks.

We briefly summarize below a series of robustness checks we performed to ensure treatment effects hold under different assumptions about spillovers of treatment or other spatial features. Here we address these.

Geospatial Spillovers.

To generate unbiased estimates of a rndomized controlled trial’s treatment effect in the presence of spillovers, we must make a theoretical presumption about how spillovers are transmitted (31). In our case, distances between villages should proxy for the relative strength of potential spillovers. Communication among residents of the study area is often face to face, so respondents from one village are likely to interact most with respondents in other nearby villages.^‡‡ The results above are robust to presuming multiple spillover distances: treatment villages 1 km, 3 km, 5 km, and 10 km away. We describe the estimation procedure in SI Appendix.

Spillovers from Interactions within Forums.

As mentioned in our description of the treatment, villages participating in the different MSF sessions were grouped together by region, so representatives from some villages in our sample interacted with each other and shared experiences. To account for this potential source of spillover effects, we created separate binary variables indicating whether each respondent attended the first, second, or third MSF. We then replicated our main analysis while controlling for those variables. The results reported in SI Appendix show that the DID estimates based on this robustness check resemble those reported here.

Refugee Camps.

Qualitative data indicate a noteworthy potential confounder that our study design did not consider: Some refugees from the ongoing conflict in South Sudan are being housed in the Albertine Graben, and the number of these refugees increased during our study. Several respondents discuss improved access to social services with enumerators, which they attribute to the creation of nearby refugee camps. There is reason to wonder whether our results are somehow influenced by household proximity to these camps.

SI Appendix, Table S3 assuages such concerns. The United Nations High Commission for Refugees was kind enough to provide geographic coordinates of all refugee camps in Uganda. We calculated the distance between each household and the closest camp. We then transformed this distance into kilometers and reran our DID estimate while controlling for household proximity to the closest camp. Our results were not significantly altered.

Data Availability.

The data and replication files have been deposited on the Harvard dataverse website and can be accessed at https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DEGFVP.

Conclusion

We find compelling short-term evidence on the efficacy of SE engagement practices in the oil and gas sector, but these findings do have some limitations: First is potential limits on external validity. We believe that many of the difficulties inherent in the oil and gas sector are similar to those faced in other renewable and nonrenewable resource extraction sectors (e.g., fears of land or resource theft, technical complexity, and weak accountability mechanisms). However, rigorous evaluations need to be replicated in different contexts and different sectors to confirm this. Second, care should be taken when replicating this intervention by different organizations. MYJ has a longstanding reputation and experience conducting face-to-face MSFs and encouraging equal participation by those who attend. The trust and openness established and facilitated in these forums may be hard to replicate with other organizations without a strong practice in community engagement, or by government- or corporate-led groups.

With these caveats, we now turn to recommendations for policy makers. Our study suggests that interacting with citizens and disseminating information increases citizens’ satisfaction with local government as well as their perceptions of transparency. This recalls findings on collaborative management and development efforts in other environmental spheres (7, 8). When transparency and civic activity increase, it follows that outcomes may improve for communities affected by extractives development (32–34). We also argue that, given sometimes pervasive misinformation about oil development in the study area, the Ugandan government can reduce public discontent by devoting more effort to ensuring that information provided to its citizens in the future is easier to understand. It may also be important that communities are given information early enough that they have time process its ramifications. Finally, our findings complement other arguments for multistakeholder environmental governance (35). Third-party organizations may be uniquely positioned to address issues of conflicting expectations among local communities, extractive firms, and national governments. These organizations could serve an important role in supporting bottom-up demands for accountable governance (34, 36). That role is worthy of more attention from future advocates for policy change.