Within-District Effects of Catholic Schooling on 12th Grad Math Achievement

Abstract

Using a propensity score matching method and regression modeling based on the 2002 Education Longitudinal Study, this study found a significant Catholic school effect on mathematics achievement among those 12th graders who were least likely to attend Catholic school. This result is evident within-districts after we used the School District Demographics System map data to locate Catholic schools within school district boundaries. Furthermore, the Catholic school effects were statistically significant for students in districts that allowed publicly funded private education.

I. Introduction

Most families in the United States who exercise school choice for their children do so by selecting better neighborhoods for their residence. Those who have the least choice are poor families with low parental education. Residing in neighborhoods with low property value, these families are often offered failing public schools that lack material and parental resources. Some argue that a solution for this educational inequality is public aid for private schooling (Figlio, 2008), which allows poor students to opt out of their low-performing public schools to attend private schools. Public aid for private schooling takes many forms, and a new development is to transform Catholic schools into charter schools. In recent years, seven Catholic schools in Washington D. C. and four in Brooklyn and Queens, New York City were transformed (Hernandez, 2009). Similar proposals have been made for Indiana and Chicago as well (Garnett & Brinig, 2011). To our knowledge, no serious evaluation of this form of public aid for private schooling has been done. The evaluation of this policy requires not only a comparison of student performance in public and Catholic schools but also a comparison of public and Catholic schools within the same school district. Using maps to match Catholic schools with public schools within the same district, our study seeks to determine whether, within districts, Catholic schools more effectively improve learning outcomes than do public schools. We also examine whether learning outcomes of attending Catholic schools within districts vary among children from different socioeconomic background. Our research will contribute to the school choice literature by gauging the district variations of the Catholic school effects.

Four questions guide our research: (1) Does a Catholic school effect exist generally for students who attend Catholic schools, compared to their counterfactuals who attend public schools? (2) Is there heterogeneity or a disproportional impact of the Catholic school effects? (3) Is there a school-district fixed effect of Catholic schooling? That is, does Catholic schooling affects students after holding constant district characteristics? Finally, (4) does the Catholic schooling effect vary across districts depending on whether they allow public funding for Catholic schooling?

We use data from the Education Longitudinal Study (ELS: 2002) to estimate Catholic school effects on 12th graders' mathematics scores. Results based on the propensity score matching method suggest that the average Catholic school effect is positive, especially for low-income and minority students and students in the districts where public aid for private schooling is available. In the next section (II), we review relevant literature, followed by a description of the data in Section III and methodology in Section IV. Section V presents the findings and Section VI concludes.

II. Literature Review

Explanations for the Catholic School Effect

The early literature on Catholic schools showed that the achievement gap by family socioeconomic status (SES) was smaller in Catholic schools than it was in public schools (Coleman, Hoffer & Kilgore, 1982; Greeley, 1982; Hoffer, Greeley & Coleman, 1985). The explanations of the more equitable Catholic school effect on achievement focused on the social relationships among parental networks within the Catholic school community. The overlapping spheres between the Catholic school and church create “social capital” among adults – the human relationships that facilitate the achievement of desirable goals (Coleman, 1988). When adults within the Catholic school community work together, there are often enforced norms that promote in high academic achievement. Such enforced norms benefit all students, regardless of their socioeconomic background.

Another major explanation for the Catholic school effect on students’ achievement is the restricted academic organization in Catholic schools, which governs students’ course-taking patterns (Bryk et al., 1993). Unlike public schools that have a wider choice of academic and non-academic courses, the curricular of Catholic secondary schools is relatively homogeneous and the core course is college preparatory. The average level of mathematics courses is higher in Catholic than in public schools, partially explaining the Catholic school effect on student achievement (Gamoran, 1996; Morgan & Sørensen, 1999; Carbonaro & Covay, 2010).

The academic benefits of Catholic schooling should be greater for disadvantaged students than for advantaged students because, without family support, disadvantaged students tend to enroll in nonacademic courses in public schools (Lucas, 1999). The highly differentiated academic structure in comprehensive public high schools tends to amplify initial social differences among students, leading to a less equitable distribution of achievement (Lee & Bryk, 1989).

However, previous findings on the effect of Catholic schooling are mixed. They depend on the methods researchers used and the outcomes measured. Whether Catholic-school advantage is attributable to the schools themselves or to the type of students self-selected to attend Catholic schools has long been the key methodological issue in the study of the Catholic school effect (Willms, 1985). Researchers have adopted several methods, such as the Heckman procedure¹, instrumental variable approach², and propensity score matching³ to address this problem of selection bias. While the effect of Catholic schooling is unclear, some researchers have found consistently positive impacts of Catholic schooling on high school graduation, college enrollment (Schwab, 1995; Neal, 1997; Figlio & Stone, 1999), and admission to top-ranked colleges (Eide et al., 2004).

The Relevance of School Districts to Catholic Schools

Funding for public schools in many states has been largely determined by local taxes collected within school districts. Although Catholic schools are not under direct control of districts, they receive funds from districts in the forms of scholarships and vouchers granted to poor students. School districts can allocate Catholic schools federal funds for disadvantaged students as well. In fact, the Title I bill authorizes local school districts to provide services to students in non-public schools who demonstrate need. Under this legislation, districts distribute federal funds to both public and private schools to provide low-performing and poor students counseling and remedial classes in reading, math, and English as a foreign or second language (see http://www2.ed.gov/policy/elsec/leg/esea02/pg1.html). In 2005–2006, about 37% of Catholic schools (versus 56% of public schools) participated in Title I (Stullich et al., 2007). Furthermore, the No Child Left Behind Act requires school districts to evenly distribute their Title I funds among poor students in participating schools, public or private schools alike. The funds to private schools are typically paid through support services. As long as Catholic schools contain low-income, racial and language minority, and other disadvantaged students, public funds are channeled to them through school districts.

Thus far we do not know whether Catholic schools affect students’ learning more or less than do public schools within school districts. An analysis of the Catholic school effect at the school district level would be appropriate and useful. If a within-district analysis finds Catholic schools to outperform public schools, converting Catholic schools into charter schools in urban school districts may be good educational practice provided that this option is viable (Hernandez, 2009).

III. Data and Method

Data Sources and Analysis Samples

This study used several sources of data. First, student-level and school-level data were obtained from the Educational Longitudinal Study of 2002 (ELS). ELS is a nationally representative sample of over 15,000 students in approximately 750 high schools. This study was based on data from the base year collected in Spring 2002 for 10^th grade, and the first follow-up collected in Spring 2004 for 12^th grade. Because of the complex and stratified sampling design, the first follow-up weight (f1qwt) was included to adjust for the design effect (see Ingels et al. (2005) for a detail discussion of the sampling strategy and research design).

Second, to supplement school-level data from ELS, we extracted school variables from the Common Core of Data (CCD) and Private School Survey (PSS). The CCD provided information about public schools, while the corresponding private school variables were from PSS. School district characteristics, such as the percent of White and Black students in a district and the district identifier, are extracted from CCD for school districts that contain public schools with 10^th–12^thgrades. PSS does not provide information about Catholic (or other private) schools’ districts. We had to use the School District Demographics System (SDDS) map data to locate Catholic schools within school district boundaries in order to extract district information for Catholic schools. All district variables were merged into the ELS sample. Finally, neighborhood characteristics (e.g., variables used to construct neighborhood segregation) came from the 2000 Census. They were then merged with the ELS students by students’ residential zipcodes.

Sample attrition between 10^th and 12^th grade affected our analytic sample. About 13% of the students exited between survey waves. Among them, about 5% were Catholic students but 88% were public school students. Dropping these cases would create sample selection bias. Our remedy is to use the multiple imputation technique (Little, 1987; Schafer, 1997) to impute missing values. The observed variables used to impute missing values include student-level information, family background, and parent and school characteristics.

Because Catholic schools are rarely found in rural areas, our sample was restricted to students attending urban and suburban schools. The full sample had about 9,240 students. In a separate analysis of the Catholic school effect within school districts, the sample was further restricted to students in districts that have both Catholic and public schools. This “district sample” includes approximately 7,470 students.

Subsequent deletion of students from the two samples was due to propensity score matching (discussed below). Two “matched samples” were created. One is the main match sample that had about 5,430 students. The other is the district match sample that had about 1,620 students.

Variables

The dependent variable is 12^th-grade standardized math score⁴, which is calibrated using item response theory (IRT). Because students’ current test scores are a result of cumulative learning through school, we need to control for students’ prior academic performance in order to evaluate the Catholic schooling effect. Our analysis takes into account 10^th-grade mathematics scores as baseline scores. In effect, we examine mathematics achievement growth from 10^th to 12^th grade. The major independent variable is Catholic school attendance, which is a dummy variable identifying whether a student enrolls in a Catholic school.

A number of demographic and family background variables are used to predict students’ school attendance in Catholic or public schools. These variables include age (in years) in grade 12; gender (male=1, female=0); race/ethnicity (Black, Hispanic, Asian, other race; reference group is White); whether student is an immigrant (1=yes, 0=not); whether English is the student’s native language (1=yes, 0=not); family composition (single-parent, step-parent, no-parent household; reference group is two-parent household); number of siblings (continuous variable from 0–6, and 6= 6 or more); working mother (part-time employee, not working; reference group is full-time employee); socioeconomic status (SES) (a continuous composite measure of parents’ highest education and occupation status); and Catholic church affiliation (1=Catholic; 0=non-Catholic).

Previous studies found neighborhood segregation to be associated with students’ propensity of opting out of assigned public schools (Lauen, 2007). In this study, neighborhood segregation is measured by the diversity index “entropy” (Reardon & Firebaugh, 2002) constructed from information on the zipcodes of ELS students’ residences⁵. Other neighborhood variables include the level of neighborhood crime reported by parents and neighborhood poverty rate (the ratio of population below poverty over those above poverty by zip codes).

We included a proxy variable measuring the number of nearby schools available for families to choose. Previous school choice studies rarely examined school supply as a factor (Downes & Greenstein (1996) is an exception). We used the Geographic Information System (GIS) technique to create the number of public schools available within a six-mile radius of the student’s residence.⁶ Apart from being a measure of school supply, this variable may also represent population density in the area where the child resides. The number of public schools nearby is likely to be greater in more populous areas.

The policy variable, public aid for private schooling, identifies whether a student lived in 2000 in a state or county that allowed for public aid to support private schooling. The information for this variable came from the 2002 State Regulation of Private School (see http://www.ed.gov/pubs/RegPrivSchl/index.html & Goodman, 2009). According to this document, Florida, Maine, Milwaukee (WI), Cleveland (OH), and Vermont had voucher programs. Iowa, Minnesota, Pennsylvania, Illinois, Florida, and Arizona instituted tax credits for private schooling, and Pennsylvania enacted a scholarship tax credit for private school students in 2001.

To examine the district variations of the Catholic school effect, we used a number of district variables. With the help of GIS and the geographical correspondence 2000 Census file (http://mcdc2.missouri.edu/websas/geocorr2k.html), school districts were linked to students’ residential zipcodes. School districts that contain Catholic schools were identified by positioning on the map simultaneously the Catholic school locations and school district boundaries. District characteristics were then extracted and attached to schools and to students within schools. District variables include the percent of receiving free/ reduced-price lunch, the percent of students who are in LEP (Limited English Proficiency) programs, the district dropout rate, the percent of White students, and the percent of Black students.

Using GIS, we created another spatial variable, cross school district, indicating whether a student attends school outside of the district in which he/she resides. Other spatial characteristics taken into account to predict students' school choice are urban school (suburban school is the comparison group) and geographical region (northeast, north central, and west; reference is south).

Descriptive Summary of the Full Student Sample

As shown in Table 1, ELS students attending Catholic schools are by and large different from students attending public schools. The majority of Catholic school students are Whites (75%). Public schools have lower proportions of Whites (58%) and higher proportions of minority students than do Catholic schools.

Table 1.

Descriptive Statistics of Student-level Variables, Full Sample

	Overall	Catholic	Public	C-P
Variables	Mean	Mean	Mean	Difference
12th-grade math score	49.18	55.31	47.75	**
Catholic school attendance	0.06	-	-	-
Previous academic performance
10th-grade math score	43.97	48.70	42.87	**
Demographics & family background
Age	18.35	18.31	18.36	**
Male	0.50	0.52	0.49
Female	0.50	0.47	0.50
White	0.58	0.75	0.58	**
Asian	0.04	0.03	0.04
Black	0.15	0.07	0.16	**
Hispanic	0.16	0.12	0.17	**
Other race	0.05	0.03	0.06
Immigrant	0.24	0.20	0.33	**
English as native language	0.81	0.92	0.78	**
Two parents	0.60	0.76	0.58	**
Step parent	0.14	0.08	0.15	**
Single parent	0.21	0.15	0.22	**
No parent	0.04	0.02	0.04	**
Number of siblings	2.30	2.00	2.37	**
Mother does not work	0.24	0.20	0.24
Mother works part time	0.19	0.22	0.18	**
Mother works full time	0.58	0.58	0.53	+
Socioeconomic status	0.04	0.46	−0.03	**
Catholic church affiliation	0.41	0.80	0.33	**
Neighborhood segregation	0.57	0.60	0.56
Neighborhood crime: low	0.73	0.78	0.73
Neighborhood crime: moderate	0.09	0.09	0.09
Neighborhood crime: high	0.02	0.01	0.02
Neighborhood poverty rate	12.27	9.77	12.85	**
# public schools within 6 miles	8.09	10.12	7.62	**
Cross school district	0.24	0.46	0.15	**
With Public aid for Catholic schooling	0.21	0.28	0.19
Spatial characteristics
Urban	0.38	0.59	0.33	**
Suburban	0.58	0.46	0.58	**
Northeast	0.20	0.27	0.18	**
Midwest	0.25	0.35	0.23	**
West	0.22	0.13	0.24	+
South	0.31	0.24	0.32	**
Number of observations	9,240	1,750	7,490

^**p<0.01,

^*p<0.05,

⁺p<0.1

Note:

¹The sample is weighted with first follow-up weight (f1qwt) and takes into account sampling design that students are nested within schools.

²Calculations based on five imputed datasets.

³According to the regulation of the use of NCES restricted data, all information related to sample size has to be rounded to 10 digits.

Compared to public school students, Catholic school students are significantly more likely to have higher socioeconomic status, to reside with two biological parents, and have fewer siblings. Catholic school students also have greater access to their mother’s time because mothers in Catholic schools are slightly but significantly more likely to work part-time than are mothers in public schools, who tend to work fulltime. Consistent with the difference in socioeconomic status, public school students are significantly more likely to live in neighborhoods with higher incidence of poverty.

Compared to public school students, Catholic school students have about two more public schools within six miles around their residences. However, they are also significantly more likely to attend Catholic schools that do not locate in the districts where they reside.

Methods

To investigate the effect of Catholic schools on student achievement, this study used the propensity score matching method (Rosenbaum & Rubin, 1983; Heckman 1998). Students attending Catholic schools are in the “treatment” group – the group of students given the Catholic education treatment. Their counterfactuals, or the “control” group, are comparable students in public schools. The goal of the propensity score matching analysis is to match students in the treatment group with students in the control group based on the characteristics related to school selection.

The matching procedure was done as follows. First, a logit model was applied to all students, including Catholic school students, to compute the “propensity” (predicted probability) of attending Catholic school. Covariates in the initial logit model included the 10^th-grade math score, student demographics, family background, spatial characteristics, and school and district variables. Because school variables were measured after any school choice decisions were made, they were excluded from the logit model that aimed to predict the choice of Catholic school. There are 60 independent variables in the final logistic regression, and the regression takes account of the design effects by estimating robust standard errors and adjusting for cluster design with sampling weight. The results are shown in Appendix Table A1.

Once students’ propensity scores were computed from the logit results, the caliper method (Dehejia & Wahba, 2002; Morgan, 2001) was used to match each Catholic school student by propensity scores with at least one but no more than five public school students. A public school student is selected only if his/her propensity score is similar to a Catholic school student’s propensity score; the difference should be no more than 0.01. The selection process produced a matched sample in which every Catholic school student has at least one matched public school student.

The sample was then arranged in five strata based on the magnitude of propensity scores. We added interaction terms (race/ethnicity or SES is interacted with all other variables) in order to achieve “balance” in each stratum, i.e., the within-stratum summary statistics of the student characteristics are similar for Catholic and public schools. After dropping the unmatched cases (“trimming”), the sample was reduced by about 41% (from 9,240 to 5,430) for the main match sample and by 78% for the district match sample (from 7,470 to 1,650).

We again checked for balance between the Catholic and public students on their demographic variables, family characteristics, and the supply of schooling (results available upon request). Most significant differences between Catholic and public school students presented previously in Table 1 disappeared. The only significant difference that remained, not surprisingly, was Catholic Church affiliation for Catholic school students.

To further check for balance between the treatment and control groups, we arranged the matched sample of students in hierarchical strata of propensity scores. This is done by performing t-tests to ensure that the average characteristics of the treatment and control groups within strata are not significantly different from each other. The standard deviation of each variable must also be similar within each stratum in order to achieve balance. The summary statistics in Table 2 show that our matched samples satisfy these requirements, suggesting that students in the same strata are alike in terms of the covariates controlled in the logit models (student demographics, family background, school supply, and interaction terms). Here students most likely attend Catholic schools are placed in the highest stratum (5), while those least likely to attend Catholic schools are in the lowest stratum (1). As expected, more Catholic school students than public school students were found in higher strata. The reverse is true for lower strata.

Table 2.

Summary Statistics of Propensity Scores by Stratum

	Catholic			Public
	Mean	SD	N	Mean	SD	N
Main Matched Sample
Stratum1	0.025	0.015	330	0.021	0.014	2130
Stratum2	0.090	0.023	380	0.086	0.023	950
Stratum3	0.185	0.031	340	0.181	0.031	410
Stratum4	0.332	0.053	340	0.314	0.051	200
Stratum5	0.532	0.081	270	0.511	0.080	80
District Matched Sample
Stratum1	0.017	0.010	70	0.010	0.010	750
Stratum2	0.063	0.017	100	0.058	0.016	170
Stratum3	0.138	0.028	100	0.132	0.028	70
Stratum4	0.262	0.044	120	0.233	0.037	40
Stratum5	0.520	0.114	220	0.485	0.132	10

Note: The number of observations is the average of five imputed data. In the main matched sample, there are 5,430 students, including 1,660 Catholic school students and 3,770 public school students. In the district matched sample, there are 1,650 students, including 610 Catholic school students and 1,040 public school students. According to the rules and regulations for the use of NCES restricted datasets, the numbers of observations reported throughout the document are rounded by 10 digits.

Next we compared the achievement outcomes of the treatment and control groups to estimate the “average treatment effect for the treated” (ATT). This is an estimate of the achievement benefit to a student who attended a Catholic school (rather than a public school). The estimate is the difference between the treatment group and the counterfactual, which can be interpreted as the causal effect of the “treatment” of Catholic schools. The ATT is first estimated by a comparison of group means without adjusting for any covariates. This is equivalent to estimating a simple regression,

$$Y_i=\alpha +\delta (C_i)+{\epsilon }_i$$ (1)

where is the 12^th grade test score for student i and C_i is Catholic school attendance. The average Catholic school effect is represented by δ.

Subsequent estimation of the ATT takes into account potential confounding factors using either the OLS (ordinary least square) model or a stratum fixed-effects model (Allison, 2005). Specifically,

$$Y_i=\alpha +\delta (C_i)+\gamma (Test{10}_i)+b_1{X}_{1i}+\dots +b_q{X}_{qi}+{\epsilon }_i$$ (2)

and

$$Y_{is}={\alpha }_s+\delta (C_{is})+\gamma (\mathit{\text{Test}}{10}_i)+b_1{X}_{1i}+\dots +b_q{X}_{qi}+\kappa Z_i+{\epsilon }_{is}$$ (3)

The covariates include the 10^th grade test score (Test10) and a host of students’ demographic characteristics, family background, and spatial characteristics, represented by X_n (n=1, …, q). Y_is is the test score of student i in stratum s, and Z is the dummy variable for each stratum. Both equations (2) and (3) estimate the net effects of Catholic school attendance, through direct control of other confounding factors, or comparison of group means among students within a smaller range of propensity scores. The resulting estimates of the Catholic school effect would be subject to less “noise.” Differences between strata that may have contributed to differences in performances between Catholic and public school students would be controlled.

IV. Results

Catholic School Effect on the Main Matched Sample

Table 3 presents various estimates of the Catholic school effect (the ATT). Models 1, 2, and 3 correspond to equations (1), (2), and (3) specified in the Method section. Without any controls, Model 1 reports the main effect of Catholic schools on 12^th graders’ mathematic scores: it is significant at the .05 level (t=5.304) and is substantively large. The coefficient of 4.72 can be translated as an overall effect size of over one-quarter of a standard deviation on the 12^th grade math test (15.23/4.72).

Table 3.

Estimates of the Catholic School Effect on 12^th-Grade Mathematics Achievement

Models	Catholic school effect	SE	t-stat	N
1: Main effect (without covariates)	4.720	0.890	5.304	5,430
2: Full model: (1)+ (10th-grade score, demographics, background, spatial)	1.222	0.515	2.372	5,430
3: Strata fixed effect	1.113	0.528	2.109	5,430
4: Stratum 1: 1st quintile (lowest)	1.312	0.632	2.074	2,460
5: Stratum 2: 2nd quintile	1.335	0.708	1.886	1,330
6: Stratum 3: 3rd quintile	0.564	0.796	0.709	750
7: Stratum 4: 4th quintile	1.940	0.943	2.057	540
8: Stratum 5: 5th quintile (highest)	−0.017	0.958	−0.017	350
9: 1st – 2ndquintiles, combined	1.282	0.547	2.344	3,790
10:3rd– 5th quintiles, combined	1.181	0.675	1.75	1,640

Note: Statistics are based on the averages of five imputed data. The matched sample includes about 5,430 students. About 1,660 are Catholic school students and 3,770 are public school students. According to the rules and regulations for the use of NCES restricted datasets, the numbers of observations reported throughout the document are rounded by 10 digits.

Design effect is controlled by taking into account school clusters and the probability weight of the first follow-up data.

However, after controlling for confounding factors in Model 2, the coefficient size dropped by about 74%, from 4.72 to 1.222 (but remained statistically significant). Thus, most of the high achievement for Catholic school students was not due to their Catholic school attendance, but rather to other factors, such as students’ family background characteristics. After accounting for propensity score strata in the fixed effect model, the effect size was further reduced to 1.113 and the statistical significance remained (Model 3). And the coefficient of Catholic schooling (1.222) is almost as large as that of SES (1.467). Judging from Models 2 and 3 together, Catholic school students do gain from their schools, and the effect size net of students’ demographic, family, and spatial factors is about 12 to 14 percent of a standard deviation of the mathematics score. It is worth noting that the achievement growth tends to be small during the last two years of high school. Students’ test score may even dropped slightly towards the end of 12^th grade after students have submitted their college applications. Therefore, the modest Catholic school effect size we found may be a conservative estimate because that effect was estimated at the point in the educational process where nothing moved test scores very much.

Model 4–8 showed stratum-specific Catholic school effect. Except for stratum 4, the ATT appears to be stronger in lower than higher strata. When adjacent strata were grouped to create two categories of high (3^rd–5^th quintiles) and low strata (1^st–2^nd quintiles), the low strata showed significant Catholic school effects but no significant effect was found within higher strata. Because students in lower strata had lower propensity scores, the result suggests that students who are less likely to attend Catholic schools benefit more from Catholic schooling than those with a higher likelihood of attending Catholic schools.

Catholic School Effect in the District Sample

Thus far, our Catholic-public school comparison has not taken school districts into account. A Catholic school student may be matched with public school students in different school districts. The next analyses created estimates of the Catholic school effect using the district matched sample in which all school districts had both Catholic and public schools. Model 1 in Table 4 shows that, overall, Catholic school students’ performance in mathematics was not significantly different from the math performance of public school students. The district fixed-effects model in Model 2 also suggests no significant average Catholic school effect within school districts. The lack of ATT in the district match sample clearly differs from the significant ATT found in the main match sample.

Table 4.

Catholic School Effect on 12^th-GradeMathematics Achievement within School Districts

	Catholic school effect	SE	t-stat	N
1: Full model	0.818	0.551	1.485	1,620
2: School district fixed effect	1.024	0.641	1.597	1,570
3: 1st – 2nd quintiles (low propensity)	1.449	0.664	2.181	1,190
4: 3rd – 5th quintiles (high propensity)	−0.376	1.681	−0.223	380
5a: With public aid for Catholic schooling	2.074	1.114	1.862	400
5b: Without public aid for Catholic schooling	0.445	0.793	0.561	1,180
6a:Strata 1–2, with aid	2.230	1.119	1.993	270
6b:Strata 1–2, without aid	1.036	0.771	1.344	920

Note: Statistics are based on the averages of five imputed data. The matched sample includes about 1,620 students. About 570 are Catholic school students and 1,060 are public school students. According to the rules and regulations for the use of NCES restricted datasets, the numbers of observations reported throughout the document are rounded by 10 digits.

Design effect is controlled by taking into account school clusters and the probability weight of the first follow-up data.

However, when the sample was divided into low and high propensity strata, in Models 3 and 4, the within-district Catholic school effect (1.449) in the lower propensity strata (1^st–2^nd quintiles) emerged as statistically significant. That is, in the same school districts, Catholic school students with low propensity scores outperform their public school counterparts with similarly low propensity of Catholic school attendance. This result on the district matched sample is consistent with the result from the main matched sample. Again, the Catholic school effect was not significant in the higher propensity strata (3^th – 5^th quintiles). Students with low propensity scores tend to come from low SES backgrounds. They were significantly more likely to benefit from Catholic school attendance than students with high propensity scores who tend to come from high SES backgrounds.

Public aid for private schooling is typically implemented at the district level. Table 4 presents two sets of district-fixed effects models to reveal if the Catholic school effect varies by public aid for private schooling. Model 5a shows a marginally significant (t=1.862, p<.10) Catholic school effect within school districts that offer public aid for private schooling. However, among students living in school districts without public aid for private schooling, Model 5b presents no significant within-district Catholic school effect.

The remaining question on the Catholic school effect was whether Catholic schooling benefitted low SES students regardless of school district policy in terms of public aids for private schooling. Models 6a and 6b were estimated on a sample that kept only students with low propensity scores (strata 1–2 combined), who tended to come from low SES homes. Model 6a revealed a statistically significant (t=1.993, p<.05) Catholic school effect for these students living in school districts where public aid was available. The mathematics scores gain was about 2.23. Model 6b showed that Catholic school students with lower propensity scores, and who did not live in districts offering public aid for Catholic schooling, did not outperform significantly their counterparts in public schools. In terms of effect size, the score gains for those living within districts with public aid for Catholic schooling was twice as large (2.230/1.036) as was the gain for those who did not live in such districts. Taken together, the results from Models 6a and 6b suggest that receiving public aid for Catholic schooling is associated with better outcomes on the math test among low-SES Catholic school students.

V. Summary and Discussions

Using data from the Educational Longitudinal Study of 2002 and supplementary data from multiple sources, we conducted a nationwide investigation on the Catholic school effects on mathematics test scores. We are well aware that there is much more to school life than mathematics. Our focus on mathematics achievement is motivated by the research finding that it is more related to in-school learning than other school subjects, such as reading. Using the propensity score matching method, we estimate academic gains for students who were enrolled in Catholic school, compared with similar students in public schools. The results suggested that, first, there appeared to be an average positive unadjusted Catholic school effect on 12-grade mathematics achievement. Catholic school students would have scored lower in mathematics had they not attended Catholic school. However, this positive Catholic school effect did not hold when we took school districts into account. Within districts, and among students with similar propensity to attend Catholic school, Catholic schooling did not differentiate students’ mathematics scores. Thus, we are unable to draw a definitive conclusion about the average Catholic school effect.

Second, there was disproportionate impact of the Catholic school effect. Students who benefitted academically from Catholic schooling were those least likely to attend Catholic schools, and also were likely to come from low SES backgrounds. This result is similar to previous studies using older data (Coleman & Hoffer, 1987; Morgan 2001). This result for students with low propensity to attend Catholic school was robust even after the school district was taken into account in the matching process.⁷

Given this empirical finding, a question arises whether policies that allow parents in low-income districts to use public monies for Catholic education may be beneficial to their children. This leads us to our third finding. There is a significantly positive Catholic school effect on math achievement within the districts where public aid for private schooling was available. Students who were least likely to attend Catholic school gained from Catholic education in districts with policies that allowed them to purchase private education with public funds. Since students who were least likely to attend Catholic schools tended to be low SES students, our result suggested that low SES students benefitted from Catholic schooling when they were eligible under some school choice policies.

The advantage of the propensity score matching method over traditional multivariate methods is that we are closer to detect causality between Catholic schooling and student achievement. However, a caveat is in order. In a random experiment, we would randomly select students and assign them to Catholic and public schools. Our method is a quasi-experiment and our focus is on the estimates of the average “treatment effect” (ATT) on the treated, i.e., Catholic school students. While we found that Catholic schooling raised test scores significant among some students, especially those who were least likely (based on their propensity scores) to attend Catholic schooling, this result does not imply that policy makers could raise the scores of a randomly chosen student if that student transferred from a public school to a Catholic school.

In light of the gradual decrease in the number of Catholic schools, a controversial proposal was made to save Catholic schools that faced closure by converting them to charter schools (Hernandez, 2009). In the last few years, some inner-city Catholic schools in Indianapolis, Washington D.C., and Brooklyn and Queens in New York were under the spotlight as they became publicly-funded charter schools (Hernandez, 2009; PR Newswire, 2011). Those Catholic schools were on the closure list because of decreasing enrolment and financial support from the Roman Catholic Church or other sponsors. Our findings may serve to support the argument to maintain these schools through public funds while also maintaining the schools’ autonomy. Such a strategy might improve disadvantaged students’ academic achievement in urban school districts. However, our findings alone are not sufficient for a recommendation on converting Catholic schools into public schools. More information is needed regarding the reasons why Catholic schooling is better than public schooling in serving disadvantaged students. If the reason lies primarily in the social capital among adults and children through their connections inside and outside of school in the Catholic community (Coleman, Hoffer & Kilgore, 1982), a new charter school that is separated from the church would not confer the same advantage to students. Alternatively, if disadvantaged students learn better in a Catholic school setting because their peers tend to be majority and high SES students – a result shown in our data, then a new charter school that is subject to public accountability for its admission or disciplinary policies governing expulsion is unlikely to produce the same type of student body that benefits the disadvantaged. That said, past research has highlighted the superior academic organization of Catholic school (e.g., Bryk et al., 1993). Future research needs to identify whether this is the most important reason for Catholic school’s success. If so, maintaining the rigorous academic structure of Catholic schools while converting the financially troubled ones into charter schools may be a good policy.

Appendix

Table A1.

Logistic Regression on Catholic School Attendance

	Coef.	Robust SE	z	P>|z|
Previous academic performance
10th -grade math score	0.01	0.00	3.45	0.00
Demographics & family background
Age	−0.11	0.08	−1.42	0.16
Male	0.16	0.08	2.10	0.04
Asian	0.71	0.70	1.02	0.31
Black	0.15	0.35	0.42	0.68
Hispanic	0.22	0.40	0.56	0.58
Other race	0.95	0.42	2.24	0.03
Immigrant	−0.25	0.17	−1.5	0.13
English as native language	0.93	0.16	5.78	0.00
Step parent	−0.50	0.13	−3.76	0.00
Single parent	−0.25	0.11	−2.26	0.02
No parent	−0.13	0.24	−0.55	0.59
Number of siblings	−0.03	0.04	−0.67	0.50
Mother does not work	0.44	0.12	3.73	0.00
Mother works part time	0.15	0.12	1.29	0.20
Socioeconomic status	0.51	0.12	4.09	0.00
Catholic church affiliation	2.41	0.11	21.59	0.00
Policy variable	0.03	0.09	0.30	0.76
Public aid available for Catholic schooling
Spatial characteristics
Neighborhood segregation	−1.25	0.39	−3.12	0.01
Neighborhood crime: moderate	0.25	0.14	1.74	0.08
Neighborhood crime: high	0.60	0.29	2.04	0.04
Neighborhood poverty rate	2.45	1.64	1.49	0.14
# public schools within 6 miles	0.04	0.01	6.24	0.00
Cross school districts	1.76	0.09	19.12	0.00
Urban	0.84	0.26	3.17	0.02
Northeast	0.38	0.12	3.12	0.00
Midwest	0.08	0.12	0.73	0.47
West	−0.44	0.13	−3.29	0.00
Interaction terms
Black * immigrant	0.46	0.34	1.35	0.18
Asian * immigrant	0.08	0.54	0.16	0.88
Hispanic * immigrant	0.17	0.27	0.64	0.52
Other race * immigrant	−0.07	0.43	−0.17	0.86
Black * SES	0.13	0.20	0.66	0.51
Asian * SES	0.26	0.22	1.17	0.24
Hispanic * SES	1.04	0.17	6.21	0.00
Other race * SES	−0.10	0.29	−0.32	0.75
Black * mother not work	−0.06	0.33	−0.20	0.84
Asian * mother not work	0.40	0.39	1.03	0.30
Hispanic * mother not work	−0.25	0.26	−0.97	0.33
Other race * mother not work	0.09	0.48	0.19	0.85
Black * mother works PT	0.67	0.35	1.89	0.06
Asian * mother works PT	−0.36	0.48	−0.74	0.46
Hispanic * mother works PT	−0.12	0.30	−0.39	0.70
Other race * mother works PT	−0.66	0.61	−1.08	0.28
Black * # siblings	−0.18	0.08	−2.18	0.03
Asian * # siblings	−0.34	0.15	−2.27	0.02
Hispanic * # siblings	−0.32	0.09	−3.61	0.00
Other race * # siblings	−0.24	0.13	−1.83	0.07
# siblings * SES|	0.05	0.05	1.17	0.24
Black * Catholic Affiliation	−0.81	0.29	−2.79	0.01
Asian * Catholic Affiliation	−0.82	0.36	−2.28	0.02
Hispanic * Catholic Affiliation	−0.11	0.35	−0.3	0.77
Other race * Catholic Affiliation	−0.50	0.40	−1.23	0.22
Urban * neighborhood segregation	1.91	0.40	4.72	0.00
Black*urban*neighborhood segregation	0.51	0.58	0.88	0.38
Asian*urban*neighborhood segregation	0.80	0.67	1.19	0.24
Hispanic*urban*neighborhood Segregation	0.53	0.46	1.16	0.25
Other*urban * neighborhood segregation	0.28	0.64	0.44	0.66
Neighborhood segregation * poverty	−9.88	2.84	−3.48	0.00
Urban* # public schools in 6 miles	−0.01	0.01	−0.98	0.51

Note: The number of observations is 9,240. According to the rules and regulations for the use of NCES restricted datasets, the numbers of observations reported throughout the document are rounded by 10 digits.

Wald Chi2= 1396.05; prob> Chi2=0.0000; log pseudo likelihood=-1462.67; pseudo R2=0.305.