Abstract
Despite substantial financial commitment to widening participation activities internationally, robust evidence demonstrating ‘what works’ in facilitating disadvantaged learners to access Higher Education (HE) is remarkably sparse. Much effort has been directed at measuring immediate post-intervention changes in the aspirations, attitudes and behaviours thought to drive access to HE, rather than actual access itself. Here, we present an innovative quasi-experimental study of a multi-intervention outreach programme (UniConnect) consisting of 1,386 learners from the Aimhigher West Midlands database whose HE application results were known, while controlling for multiple variables, including estimates of deprivation. The results showed that any engagement with UniConnect, no matter how limited, was associated with an improved chance of achieving a place in HE, but the type of engagement, the extent of engagement and the combination of types of engagement all mattered. The more learners engaged with UniConnect, the greater were their chances of HE acceptance, but the benefit of each additional engagement beyond five or six engagements was small. To our knowledge, these findings are the first to indicate the number, type and combinations of interventions that are most effective in supporting progression to HE. These results therefore have important implications for future practice, enabling funding for such work to be used for optimal impact. Furthermore, we found large differences in success between schools, even when controlling for several other variables; a finding which has important implications for future evaluation research.
Keywords: Access and participation, Evaluation, Higher education, Widening participation, Multi-intervention, UniConnect
Access and Participation; evaluation; Higher Education; widening participation; multi-intervention; UniConnect.
1. Introduction
Across the world, there are persistent socio-economic and demographic based inequalities in terms of educational qualifications. These inequalities can have a detrimental impact on later life chances in terms of employment, wealth, health and housing [1, 2, 3, 4]. James et al. [5] concluded that internationally there are ‘persistent inequalities in educational participation and outcomes, with major social inequities to higher education in particular, despite mass education systems’ (p1) (see also [6]). Regardless of educational achievement, there are also different expectations of entering tertiary education amongst children of parents in higher versus lower status occupations, although educational inequalities differ across countries, suggesting that these inequalities are mutable [7]. In the UK, evidence suggests that inequalities are increasing, with more than one in five of the population living in poverty [8]; an increase of 12.5% over 5 years. Successive governments have attempted to address these inequalities through policies and funding to improve social mobility. A focus of such policies has included attempts to improve disadvantaged groups' lower progression rates into Higher Education (HE); that is education beyond secondary level, most commonly offered at a university or higher education college. These attempts are often in the form of widening participation or ‘outreach’ programmes that aim to increase expectations and intentions, attainment, attitudes, awareness, and knowledge. In the UK, university outreach teams have driven such initiatives under requirements and regulations set out by the Office for Students (the HE regulator). Resource allocations to these initiatives are large, and so the stakes are high; the UK Government anticipated spend on widening participation by the HE sector in 2020–21 to reach around £860m [9]. However, given the amount of resource historically and currently allocated to these activities, robust and objective evidence on ‘what works, under what circumstances and for whom’ is remarkably sparse, Skilbeck arguing that, ‘a significant limitation in drawing general conclusions for future action on the basis of ‘international good practice’ is the dearth of evaluative research…’ [6].
Gorard and colleagues (p32) [10] conducted an extensive and far reaching review of widening participation research in England, but found that substantial proportions of the literature had to be excluded from the review on the basis of either quality of evidence or inadequate reporting, concluding that research on the efficacy of interventions is ‘a major blindspot for the whole field’ (see also [11, 12, 13, 14]). Changes in attitudes, aspirations, knowledge and behaviours are frequently cited as evidence of impact of interventions, but in a review of almost 170,000 pieces of evidence, Gorard et al. [15] found little evidence of a causal link between attitudes to education and either attainment or participation, although an association was confirmed. Bergin et al. [16] found that interventions were more likely to affect which institution participants attended, than whether they attended at all. Robinson and Salvestrini [17] provided a helpful updated review of evidence of the impact of various widening access initiatives internationally, but progress has been slow. They noted: (i) the challenge of inferring from these evaluations which components of multi-intervention programmes have led to any success - a difficulty that hinders generalisation of the results to other programmes; (ii) the lack of evidence on the impact of actual enrolments, as opposed to reported changes in aspirations and attitudes; and (iii) the lack of evidence demonstrating causality. In their review, Robinson and Salvestrini [17] categorised only three studies evaluating multi-intervention outreach as providing evidence of causality (one finding positive impact); the Transforming Access and Student Outcomes (TASO) website lists four, just two of those reporting positive impact. First, in the US, Bowman et al. [18] reported positive and significant effects of the GEAR UP programme on enrolments and graduation. Second, in the UK, Emmerson et al. [19] found a positive impact of an Aimhigher programme on HE participation rates across Local Education Authorities that did and that did not participate, although the effect found was not statistically significant overall and the interpretation of the results was hindered by a difficulty in disentangling results from a separate programme.
Following the removal of funding for a national programme of coordinated outreach, known as Aimhigher, the University of Birmingham, Aston, Birmingham City University, University College Birmingham and University of Worcester continued to collaborate with a model based on partnership subscriptions and targets linked to their Access Agreements (now Access and Participation Plans). This partnership is known as Aimhigher West Midlands (Aimhigher WM) and conducts outreach activities across 25 rural and urban West Midlands' wards. Aimhigher WM's UniConnect programme (formerly known as the National Collaborative Outreach Programme: NCOP) was established to support the government's social mobility goals of increasing the number of young people from underrepresented groups who go into HE, from wards where participation was lower than might be expected given the GCSE results of the young people who live there. The UniConnect programme aims to address this unexplained ‘participation gap’ via increasing confidence and motivation to succeed at school; expectation and intention to progress to HE; awareness and knowledge of HE; and attainment and learning in Key Stages 4 and 5.
Morris and Golden [20] previously summarised research on the impact of the national Aimhigher programme by a number of authors and suggested that although there was evidence of the programme impact on GCSE gains, there was no statistical evidence that it changed participants’ minds about going on to HE (although there was qualitative evidence that it may have widened the horizons of certain groups). Chilosi et al. [21] evaluated the effects of an Aimhigher programme on GCSE attainment, HE applications and HE entries. They overcame reported ethical and legal concerns regarding tracking of pupils by using multiple regression analysis on cohort level (rather than individual) data and reported a positive effect of Aimhigher on all three measures overall, although they also suggested that the programme may not have had the desired effect of increasing HE entry in pupils from lower socioeconomic backgrounds. The present report builds on previous work [22] evaluating the effectiveness of UniConnect to provide a more innovative and statistically rigorous evaluation of the impact of UniConnect interventions on the rate of successful Universities and Colleges Admissions Service (UCAS: who operate the application process for all British Universities) acceptances. More specifically, our analyses enabled us to estimate the contribution of the different interventions that formed the intervention programme, indicating which number, type and combinations of interventions are most effective in supporting progression to HE. In our methods section, we outline the interventions implemented, data collected, variables used and the participant cohort and provide a summary of our general analysis strategy. In the results we detail the outcomes of that strategy to provide information on: (i) the impact of Uniconnect engagement, (ii) the most effective types and combinations of Uniconnect engagement and (iii) the combination of other factors associated with UCAS success. Finally, in the discussion we consider more general learnings and recommendations from the data for optimising multi-intervention outreach programmes, as well as potential limitations of this study.
2. Materials and methods
2.1. Design
This was a retrospective, quasi-experimental study investigating the extent to which engagement with UniConnect activities was associated with successful application to HE. All learners included in the study were eligible to participate in UniConnect activities but varied in the extent to which they did so, including some who did not participate at all. This allows us to examine the relationship between the number and type of UniConnect activities participated in and the outcome of successful application to HE. Participants were not randomly allocated to different levels of UniConnect intervention; instead, the degree of engagement was determined by a combination of the learners' and the schools’ choices. For example, those who did not engage with UniConnect at all, may have simply chosen not to out of lack of interest, or their school may have chosen not to offer them the opportunity, perhaps because they were deemed to have insufficient academic ability.
The critical outcome measure was UCAS application success, meaning that the learner had been accepted onto a course of prescribed HE that included HNDs, HNCs, foundation degrees, a degree or degree or graduate level apprenticeship. Learners classified as ‘unsuccessful’ included those who made unsuccessful UCAS applications as well as those who made no application at all. There were three categories of independent variables included in this study: Participant-related, School-related and UniConnect Intervention-related.
Each partner involved in the Aimhigher consortium recorded their own data on the Aimhigher tracking database. This database holds data on pupils’ background characteristics and is employed to track pupils’ engagement within interventions. UCAS data was obtained directly from schools via a standardised excel sheet and was then matched to records from the Aimhigher database to allow us to explore whether there was an association between HE outcome, frequency of engagement and intervention type(s).
2.2. Participants
Participants in this study were drawn from a population of 2,706 18-19-year old learners completing full time Level 3 qualifications selected from the West Midlands UniConnect database who were due to make a first application to university in the 2017/8 or 2018/9 application cycles. The sample does not include individuals in this age group who were on other career pathways such as completing part-time Level 3 qualifications, re-taking Level 2 qualifications, completing an apprenticeship, in employment or training. For this reason, the overall HE participation rates of the UniConnect population will be lower than those stated here.
The sample considered here consisted of 51% of this larger cohort (n = 1,386, 792 women, 57.1%) selected on the basis that the results of their UCAS applications were known. A selective sampling approach was employed, where data was requested from schools that had large numbers (100 plus learners) or proportions (20%+) of UniConnect learners on roll and which received high levels of UniConnect resources and funding; this included a payment for teachers to help to coordinate activities, access to mentoring and tutoring and an Aimhigher WM ambassador working in the school or college with the pupils. Data was returned for 40 out of 46 schools in the 2017/18 cycle and 32 out of 40 in the 2018/19 cycle. This provided a return rate across both years of 81% of schools. The sampling approach meant that the learners mostly attended larger schools but were otherwise broadly representative of the larger cohort in terms of age and ethnicity. Seven hundred and eighty-six (57%) learners applied to HE in the 2017/18 UCAS cycle and 600 (43%) in the 2018/9 cycle.
Individual ages were not available, but most were in Year 13 of school or their 2nd year at College (n = 1306, 94%) when they engaged with the UniConnect programme meaning they would have been 16–18 years old. Nearly three quarters of the learners self-identified as white (n = 1009, 72.8%), mostly White-British (59.8%), 21.5% identified as Black or minority ethnic status (BAME) and 6% declined to define their ethnicity. The largest BAME groups were 3.4% Black-British (Caribbean), 2.9% Black-British (African), 2.7% Mixed (white-Caribbean), 2.5% Asian-British (Pakistani), 1.9% Asian-British (Indian), 1.7% mixed (white and Black-African) with other ethnicities making up less than 1% of the sample.
Participants’ area of domicile was identified by the Census Area Statistics (CAS) ward in which each learner lived. CAS wards are small local areas used in the 2001 census that contain, on average 5,500 people, although this varies widely. CAS wards included in this study were rated as POLAR3 Quintile 1, meaning they were in locations where the rate of participation in HE was in the lowest quintile in the UK, with an average of 16.1% of all young people going to a University or FE College compared to a national average of 37.4% [23].
Estimates of individual deprivation were derived from the 2019 English Indices of Deprivation measures (IoD2019) [24]. This is a post (zip) code measure of disadvantage. In addition to a measure of overall deprivation, IoD2019 provides estimates of deprivation by locale in seven different domains: Income, Employment, Education, Health, Crime, Barriers to housing and services and the Living Environment) and the supplementary index of Income Deprivation Affecting Children Index (IDACI) [25]. The IoD2019 and IDACI were available as rankings (from 1, most deprived to 32,844, least deprived), deciles and, in some cases, raw scores (Income, Employment, IDACI). The learners predominantly came from relatively deprived areas. The median and lower and upper quartiles scores on the IoD2019 and IDACI are shown in Table 1. As can be seen, the median scores on most measures placed these areas around the 12th percentile of all districts in England, although they did rather better on ‘Crime’, ‘Barriers to Housing and Services’ and the ‘Living Environment’. In raw figures, 25% of the families in these areas experienced deprivation relating to low income, the unemployment rate was 18% and nearly a third (31%) of children lived in income-deprived families.
Table 1.
Deprivation Index | Ranking (out of 32,844) |
Percentile |
Raw Score |
||||||
---|---|---|---|---|---|---|---|---|---|
Q1 | Median | Q3 | Q1 | Median | Q3 | Q1 | Median | Q3 | |
Index of Multiple Deprivation | 1,988 | 4,082 | 9,392 | 6.1 | 12.4 | 28.6 | |||
Income | 1,855 | 3,999 | 10,161 | 5.6 | 12.2 | 30.9 | 15.4% | 25.0% | 31.3% |
Employment | 1,867 | 4,215 | 10,534 | 5.7 | 12.8 | 32.1 | 11.4% | 18.1% | 23.1% |
Education and Skills | 2,126 | 4,503 | 8,949 | 6.5 | 13.7 | 27.2 | |||
Health and Disability | 2,871 | 5,142 | 9,874 | 8.7 | 15.7 | 30.1 | |||
Crime | 5,289 | 9,296 | 14,191 | 16.1 | 28.3 | 43.2 | |||
Barriers to Housing and Services | 5,479 | 9,176 | 15,637 | 16.7 | 27.9 | 47.6 | |||
Living Environment | 5,766 | 11,730 | 20,160 | 17.6 | 35.7 | 61.4 | |||
Income Deprivation Affecting Children Index (IDACI) | 1,549 | 3,956 | 10,223 | 4.7 | 12.0 | 31.1 | 19.8% | 31.1% | 39.5% |
2.3. Schools
For convenience, the term ‘school’ is used here to include both schools and FE colleges. Individual data on prior educational achievement was not available for individual learners, although all had been on a Level 3 course before the UniConnect programme was launched which means they must have achieved a good level of Key Stage 4 (GCSE) attainment. Learners came from 42 different schools out of the 81 UniConnect target schools with an average of 40 individuals each although the numbers varied widely (SD = 54.4; range: 1–270). Five schools accounted for 46% of the total, each with more than 70 learners each, but 22 schools had fewer than 20 learners each and six had fewer than ten. Information on each school was available including the number of learners, the UCAS success, the average ‘A’ level performance (mean ‘A’-level points achieved, progress, percentage achieving AAB grades, average best grade, etc.), learner destination (HE, employment, apprenticeships), Office for Standards in Education (Ofsted) assessment, UniConnect engagement and mean deprivation of the IoD2019 and IDACI scores of the individual learners that attended them.
2.4. UniConnect interventions
The models of delivery varied between rural and urban schools, but all interventions were classified into seven different types of activities as shown in Table 2, which also shows the standard duration of each type of activity and the number of times that each was delivered (by either number of pupils or number of programmes). Of the 1,386 learners, 955 (69%) engaged in at least one UniConnect activity. The mean number of engagements was 2.9, although the distribution was very skewed with most users engaging on one or two occasions (Mdn = 1). However, a small number of individuals engaged frequently, with the top 1% engaging more than ten times each. The most common form of engagement was seeking information, advice and guidance (information and guidance: 44%) followed by master classes (30%), mentoring (21%), campus visits (9%), tutoring (3%), summer school (2%), work experience (<1%) and other (<1%). As work experience and other activities were so rare, involving around 1% of all learners, they were excluded from all further analysis. All activities included some degree of information, advice and guidance. A Venn diagram showing the co-engagement of the five most types of UniConnect activity is shown in Figure 1 [26]. With six different UniConnect activities, there are 63 possible combinations of UniConnect activities, excluding no activity. However, most individuals (94%) fell into one of only twelve combinations. It should be noted that for all the data presented in this report, there is likely to be an element of self-selection bias, as pupils with different demographics, socioeconomic background and prior attainment characteristics were more likely attend certain interventions than others. In practice, access to many of the activities were organised through the schools and the extent to which these activities were truly accessible to all learners varied according to local practice. This means that those learners who did not engage with UniConnect probably form a heterogeneous group that includes some who were uninterested in engaging at all, some where the school did not encourage or allow engagement and others who may have engaged had the opportunity been made available.
Table 2.
Activity | Delivery Period | Duration | Frequency | # times activity delivered | Time of year |
---|---|---|---|---|---|
Information Advice and Guidance | Throughout year | 0.5–7 h (average 2 h) | Throughout year | 1690 | Sep–Jul |
Masterclass | Throughout year | 1–7 h (average 3 h) | Throughout year | 699 | Sep–Jul |
Mentoring∗ | 40 weeks | 19 h | Weekly | 6 (programmes) | Sep–Jul |
Campus Visits | Throughout year | 2–7 h (average 4.45 h) | Throughout year | 296 | Sep–Jul |
Tutoring∗ | 20 weeks | 10 h | Weekly | 2 (programmes) | Sep–Jul |
Summer School | 2–3 days | 20–30 h | Annual | 34 | Mar–Apr or Jun–Jul |
Community Based Interventions | 40 weeks | 1–5 h (average 2 h) | Weekly | 1 (programme) | Sep–Jul |
Both mentoring and tutoring sessions typically last for 1 hour.
There are five universities within the urban area, and each provided two members of staff to support the co-ordination of activities within schools. Recent graduates known as UniConnect Progression Ambassadors were placed within embedded schools to deliver support for learners (mentoring, information and guidance, and workshops for learners and parents/carers) and to facilitate their access to activities delivered by the partner universities. In addition, these schools were provided with additional funding to help appoint a member of staff to build capacity to support the UniConnect programme. In rural areas, co-ordinators were linked to schools to support the school's participation with UniConnect. Rather than UniConnect Progression Ambassadors, Graduate Ambassadors and FE mentors were commissioned to visit schools and offer online support. In these rural areas, a commissioning model was run in which schools bid for funding to deliver activities to meet the needs of learners within their organisation which they would not otherwise be able to afford to provide.
2.5. Statistical analysis
The dependent variable in all analyses was UCAS success (Yes/No). Identifying predictors of a binary dependent variable was conducted using direct Logistic Regression using the χ2 test of significance for the overall model, with -2 log likelihood ratio (LLR), the Cox & Snell pseudo-R2 (CSR2) and case classification (including sensitivity and specificity) as indices of the completeness of the model and for comparison between models. Low -2 log likelihood ratios and Cox & Snell pseudo-R2 values approaching 1 indicate better fit to the data. High χ2 values are also associated with better fit but values can only be compared when they have the same numbers of degrees of freedom. Sensitivity is the true positivity rate, in this case, the percentage of individuals who were predicted to achieve UCAS success out of all of those who did. Specificity is the true negative rate, in this case, the percentage of individuals who were predicted to have failed to achieve UCAS success, out of all those who failed. Sensitivity and specificity rates of at least 80% are usually required to be useful, although this very much depends upon context. The importance of individual independent variables was assessed using the odds ratio, Exp(B), with 95% confidence intervals as the index of significance. The odds ratio is the ratio of the odds of the successful HE application in one group (odds being the number of people who successfully applied to HE divided by the number who were not successful) to the odds of the positive outcome in the other group.
In the case where there was a single dichotomous independent variable, relative risk was used as the index of importance instead of the odds ratio. Although odds ratios are widely used, notably in Logistic Regression, they are commonly and erroneously misinterpreted as relative risks. Relative risk is the ratio of the probability of the successful HE application in one group to the probability of the successful HE application in the other group, so for example, a relative risk of 1.5 would mean that the group is 1.5 times, or 50%, more likely to have a UCAS acceptance than the other group. The RRI's major advantage over the alternative measures of UniConnect engagement is that it better reflects the relationship between engagement and the chance of progressing to HE. It is also simple to derive from the number of engagements along with an estimate of the associated probability of progressing to HE. As relative risks are more intuitive to understand than odds ratios, they were used in preference whenever feasible.
2.6. Ethics statement
The Aimhigher West Midlands programme has obligations set out by the Office for Students to identify what interventions are most effective for the public benefit in terms of closing gaps in school and higher education inequality. No new or additional data were collected for this research and all data were anonymised.
3. Results
3.1. The impact of UniConnect engagement
Learners who engaged with UniConnect activities were much more likely to progress successfully to HE (58%) than those who did not engage (39%). This means that any engagement with UniConnect, no matter how limited, was associated with an improved chance of achieving a place, giving a relative risk of 0.58/0.39 or 1.49. In other words, those who engaged were nearly 50% more likely to be accepted into HE than those who did not (95% CI [1.31, 1.70]). This effect, although highly statistically significant, was small (χ2df=1 = 43.1, p < .001; LLR = 1875.4; CSR2 = .031) improving the correct classification of success to 58.9% from a baseline correct classification rate of 51.9%. The sensitivity of 57.9% and specificity of 61.3% were also poor.
It was considered that better classification might be achieved by combining information from across all UniConnect activities and by using the total number of UniConnect engagements, rather than a simple measure of engaged/not engaged. Using the total number of UniConnect engagements was found to be a significant predictor of UCAS success (χ2 = 29.24, df = 1, p < .001; LLR = 1890.1; CSR2 = .021), but again, the association was weak with 58.9% of cases correctly classified compared to the baseline correct classification of 51.9% (sensitivity 55.1%, specificity 62.9%). Note that the higher LLR and the lower CSR2 suggest that the total number of UniConnect Engagements is a poorer predictor of UCAS success than the simple binary measure on engagement.
One reason for the poorer prediction of the total number of UniConnect engagements (i.e. the Total Score) is because it assumes a linear relationship between engagement and UCAS acceptance, such that the more individuals engaged with UniConnect, the more likely they were to achieve UCAS acceptance. However, most relationships of this type are governed by a law of diminishing returns, whereby each increase in activity provides a smaller additional effect until an asymptote is reached where no further benefit is gained no matter how much the activity is increased. In order to model this relationship we estimated the probability of UCAS success at different levels of UniConnect engagement. As few individuals engaged with more than a small number of UniConnect activities, we averaged across numbers of engagements to ensure sufficient sample size in each bin to get a stable estimate of the response. Specifically, we estimated the relative risk of UCAS success at activity levels of 1, 2–3, 4–5, 6–7, 8–9, 10–11, 12–13, 14–15, 16–17 and ≥18 engagements, compared to no engagement, weighted by the number of individuals in each bin and fitted a Brody curve, (a commonly used monotonic growth function with easily interpretable parameters): see Figure 2. The curve provided a good fit to the rata (adjusted R2 = .90; RMSE .09) that reached asymptote at a relative risk value of 1.69 meaning that no matter how much UniConnect engagement learners have, they should not expect to improve their chances of UCAS success by more than around 70% above those who did not engage.
This non-linear relationship between the number of UniConnect Engagements and UCAS success provides a convenient way of estimating each individual's likely benefit from their engagement with UniConnect which we call the Relative-Risk Index (RRI), also tabulated in Figure 2. This shows that engaging in a single activity raises the RRI from 1.00 to 1.25 and raises the probability of progression to HE from 39% to 49%. In contrast, engaging in seven activities vs. six activities changes the RRI from 1.64 to 1.66 and the probability of progression to HE from 64% to 65%. At its greatest, engaging with UniConnect provided a nearly 70% greater chance of UCAS acceptance than someone who did not engage. To get this full benefit however, more than a dozen engagements might be required but 90% of the maximum benefit was could be expected with as few as five or six engagements. Using logistic regression, the RRI was found to be a significant predictor of UCAS success (χ2 = 59.4, df = 1, p < .001; LLR = 1859.6; CSR2 = .042). However, the association remained weak, with 58.9% of cases correctly classified compared to the baseline correct classification of 51.9% (sensitivity 55.1%, specificity 62.9%). Nevertheless, the RRI performed substantially better as an index of the degree of UniConnect engagement than either the total number of UniConnect engagements or binary measure of UniConnect engagement and so was used in further analyses (based on a higher CSR2 value and a lower LLR value).
3.2. The most effective types and combinations of UniConnect interventions
The different types of engagement with UniConnect were not all equally effective. Figure 3a shows the relative risk of UCAS application success by activity type (whether engaged with alone or in combination with other activities) and it can be seen that the UniConnect activities most strongly linked to UCAS acceptance were summer schools, campus visits and information and guidance whereas tutoring offered no significant benefit.
Similarly, not all combinations of engagement types were equally effective (see Figure 3b). Here, summer schools and combinations of information, campus visits and master classes were most effective. The predictive value of the 14 most common combinations of UniConnect activity together was explored using logistic regression with the 14 combinations of UniConnect engagement entered as a categorical independent variable with ‘No engagement’ as the reference category. The resulting model was statistically significant (χ2 = 68.75, df = 14, p < .001; LLR = 1850.6; CSR2 = .048) with 59.1% of cases correctly classified and relatively good levels of sensitivity (74.6%), although the specificity was poor (42.3%). Learners who engaged with a single type of UniConnect activity tended to be less successful that those who engaged more widely. Combinations of activities that included summer schools did particularly well (see Figure 3b), with the second-best combination being information and guidance, master classes and campus visits.
To summarise the results so far, any UniConnect engagement was associated with substantially better chance of UCAS success but the type of engagement, the extent of engagement and the combination of types of engagement all mattered.
3.3. Combination of other factors associated with UCAS success
So far, we have considered the impact of UniConnect interventions in isolation and ignored other potential influences on UCAS success. However, to provide a comprehensive evaluation of the effect of UniConnect interventions, we need to consider their effects in combination with other possible influences on UCAS success. One way to do this would be to expand the logistic regression analyses to include other independent variables of interest (e.g. demographics, school, levels of deprivation) but there is good reason to suspect that this approach would be suboptimal as all the learners were nested within schools and different local communities, each of which is likely to have a significant effect on UCAS success. In such cases, a multi-level analysis with categorical outcomes is appropriate and we adopted this approach following the analysis strategy recommended by Heck et al. [27]. In the following analysis, the contribution of the relevant independent variables to UCAS success was estimated using robust multi-level logistic regression with UCAS acceptance (Yes/No) as the dependent variable (IBM SPSS 26).
3.3.1. Level 1 fixed effects
The independent variables were (with a brief rationale for their inclusion) as follows:
Sex and Ethnicity. Women are more likely to attend university in the UK than men [28]. Black, Asian, and ethnic minority learners are more likely to enter HE than white learners, particularly amongst lower SES communities. White ethnic groups made up 76.8% of those in HE but 84.6% of the overall population of England and the proportion of white students fell by 37% between 2002/3 and 2017/18. Overall, low SES white men have a significantly lower rate of university attendance than white women, or men from BAME (Black, Asian and Minority Ethic) communities. For these reasons, the interaction between sex (men/women) and ethnicity (white/BAME) was entered into the analyses.
Deprivation Indices. Deprivation is associated with lower educational outcomes. The association between deprivation indices and UCAS success was investigated using the IoD2019 index of overall deprivation. As the rankings scores were skewed, we used the log of the ranks. We included the IoD2019 overall index of deprivation but, in addition, as we wished to explore the specific contributions of each of the IoD2019 subscales (Income, Employment, Health, Crime, Barriers to Housing & services, Local Environment) and the IDACI. To do this, we regressed each subscale onto the overall index score in turn and estimated the residual scale for each. These residual scales provided an estimate of the unique variance for each subscale (i.e. the variance not shared with the overall index) and were independent of each other, thus avoiding problems of multicollinearity.
UniConnect Engagement. Although the precise combination of UniConnect Activity provided the best predictor of UCAS success, we elected to use the RRI of UniConnect Engagement as this performed nearly as well as the combination measure but had the simplicity of being a simple and easily estimated index that could be applied to all learners.
Rural/Urban. The type of location of the school (rural vs. urban) was included as the structure of UniConnect interventions differed between rural and urban schools.
3.3.2. Level-2 random effects
All learners and UniConnect interventions were nested within schools and locations and for this reason both factors were considered as candidates for Level-2 random variables in the model.
We first considered school. Essentially, this involved determining whether the variation in outcomes for learners in different schools was sufficiently large to make including school as a random effect in the model worthwhile. This was done by producing a multi-level model of UCAS success with a single Level-2 random effect (i.e. school). The results of this analysis showed that the school attended was a significant predictor of outcome (Odds ratio = 1.444, t = 2.989, p = .003) and that the variance between schools was significantly large (variance of the intercept = .326, z = 2.680, p = .007), accounting for approximately 9% of the variance in outcome.
Area of domicile is an important indicator of the likelihood of entering HE. As UniConnect interventions were focussed on CAS wards where learners were least likely to attend university, (quintile 1 of POLAR3), we used CAS wards as the identifier of area of domicile, but, when this was tested, the model was not significant (Odds ratio = 1.101, t = 1.248), p = .212) and the variance between CAS wards was not significantly large (variance of the intercept = .056, z = 1.3.06, p = .192), so this was not included in the final model.
3.3.3. Final model
As data on ethnicity had been refused by 79 individuals, the sample size for this analysis was 1,307 (94.3% of the total), of which 67.1% were correctly classified in terms of their UCAS success (sensitivity 72.0%, specificity 61.8%). The results of the Level-1 fixed effects - after inclusion of the school variable as a random effect - are shown in Table 3 and described below.
Table 3.
Fixed Effects | Coefficient | S.E. | t-value∗ | p-value | Odds ratio | 95% C.I. |
|
---|---|---|---|---|---|---|---|
Lower | Upper | ||||||
Intercept (School) |
-2.11 |
0.55 |
-3.87 |
<.001 |
0.12 |
0.04 |
0.35 |
Sex∗ BAMEa | |||||||
BAME women | 0.91 | 0.25 | 3.70 | <.001 | 2.49 | 1.53 | 4.04 |
White women | 0.26 | 0.12 | 2.26 | 0.024 | 1.30 | 1.04 | 1.63 |
BAME men |
0.87 |
0.22 |
3.98 |
<.001 |
2.38 |
1.55 |
3.65 |
Index of Deprivation |
0.02 |
0.10 |
0.19 |
0.849 |
1.02 |
0.85 |
1.23 |
Income | 0.58 | 0.94 | 0.62 | 0.535 | 1.79 | 0.29 | 11.21 |
Employment | -1.11 | 0.63 | -1.77 | 0.077 | 0.33 | 0.10 | 1.13 |
Education | -0.12 | 0.33 | -0.35 | 0.726 | 0.89 | 0.47 | 1.70 |
Health and Disability | -0.63 | 0.41 | -1.56 | 0.120 | 0.53 | 0.24 | 1.18 |
Crime | -0.25 | 0.34 | -0.74 | 0.458 | 0.78 | 0.40 | 1.51 |
Barriers to Housing and Services | -0.16 | 0.37 | -0.42 | 0.676 | 0.86 | 0.41 | 1.78 |
Living Environment | -0.55 | 0.32 | -1.71 | 0.087 | 0.58 | 0.31 | 1.09 |
IDACI | -1.28 | 0.47 | -2.74 | 0.006 | 0.28 | 0.11 | 0.69 |
UniConnect Engagement (RRI) | 1.41 | 0.40 | 3.53 | <.001 | 4.10 | 1.87 | 8.99 |
RRI by Rural/Urbanb | 0.50 | 0.23 | 2.17 | 0.030 | 1.65 | 1.05 | 2.59 |
df = 1269;
Reference Category: white man;
Reference Category: Urban.
Sex and Ethnicity. BAME women (relative risk = 1.41), white women (relative risk = 1.15) and BAME men (relative risk = 1.43) were all much more likely to attend university than white men. Women were significantly more likely to achieve a university place than men (relative risk of 1.10; 95% CI [1.04, 1.16]) which is somewhat lower than the national average where 30% more women than men currently attend HE. If we consider the seven largest ethnic groups in this sample, three showed substantially greater UCAS success than white learners (Asian British-Indian, Asian British-Pakistani and Black British-African) with relative risks of UCAS success of 1.51, 1.65 and 1.55 and only one group (Mixed White and Black Caribbean) were significantly less successful than their white counterparts (risk ratio = 0.65).
Indices of Deprivation. Overall, UCAS success was not associated with deprivation. Only the IDACI was significantly associated with UCAS success (Odds ratio = 0.28, t1269 = -2.74, p = .006). Note that the predictor was not the IDACI score per se but the residual of the IDACI score regressed on to the IoD overall deprivation score. That is, learners living in areas where the proportion of children affected by income deprivation was higher than would be expected (i.e. more deprived), given the overall level of deprivation in that area, were slightly more likely to achieve UCAS success.
UniConnect Engagement. The relative-risk index of UniConnect engagement remained associated with increased probability of UCAS success (odds ratio 4.10; 95% CI [1.87, 8.99]) even with other factors (school, sex, ethnicity and deprivation) considered. The interaction between RRI and Rural/Urban location was also significant (odds ratio 1.65; 95% CI [1.05, 2.59]) suggesting that there was a difference in effectiveness of UniConnect interventions in favour of rural locations.
School. As already, noted, School was a significant random factor, but with the addition of the fixed effects, the proportion of variance accounted for fell slightly to 9.6% (variance of the intercept = .350, z = 2.34, p = .019). This made school a substantially better predictor of UCAS success than any of the measures of UniConnect engagement discussed. Figure 4 shows the relative risk for each school compared to all other schools. These showed a very wide range from the least successful school, where learners have less than a third of the chance of entering HE in comparison to learners at other schools (relative risk = .32), to the most successful school where learners were nearly twice as likely to be successful (relative risk = 1.94). In terms of UCAS success rates, and ignoring schools with fewer than ten UniConnect learners, the rate of success across schools ranged from 17% to 88%.
Given the importance of school in UCAS success rate, as a control analysis, we investigated if this success rate was associated with variations between schools in UniConnect engagement – it was not. Similarly, we tested whether variation in success rate was associated with variation between schools in terms of levels of deprivation. In this case, there was some evidence that schools with a higher proportion of learners living in areas with greater education and skills deprivation than would be expected given the area's overall level of deprivation, tended to have lower UCAS success rates, although it accounted for less than 8% of the variation. Overall, therefore, the variation in success rates observed between schools is not accounted for by any of the variables that we measured.
4. Discussion
The primary finding of this study is that engagement with UniConnect interventions was associated with a higher probability of being accepted into HE. Although the type of engagement, the extent of engagement and the combination of types of engagement all mattered, any engagement, no matter how modest, significantly enhanced the learner's chance of UCAS success. This was true even when other factors, like sex and ethnicity, the school attended, rural vs. urban environment and the level of deprivation were statistically controlled. Although this finding emerges from the UK, the fact that it applies across such a broad range of conditions within the UK suggests it is likely to be applicable in other contexts also.
Emmerson et al. [17] reported positive effects of an Aimhigher programme, with greater effects on pupils from disadvantaged backgrounds. Our results reaffirm this finding in individual pupils from disadvantaged backgrounds, providing a robust statistical analysis and controlling for several critical factors, including school and local area. We have also extended the finding to include information on which components and combinations of components of a programme have the greatest impact on access to HE. The best combinations of activities for improving outcomes in our analysis also included summer schools; although combinations of information and guidance, master classes and campus visits were also effective. Previous research has suggested that summer schools are amongst the most effective interventions, although not necessarily the most cost effective [29, 30, 31].
Our results also show how between five and six components in a multi-intervention programme provide the optimal balance between input and impact, although simply having engaged with UniConnect at all was the single best predictor of UCAS success. Although it is generally accepted that multi-intervention programmes are more effective than single interventions (see e.g. [14, 17]), to our knowledge there has been no previous research on the necessary, sufficient or optimal number of interventions; our findings address this gap.
It was interesting to note that our results showed that the school attended was a better predictor of UCAS success than any measure of UniConnect engagement. Chowdry et al. [32] also noted the potentially important role that schools seem to play in encouraging pupils from lower socio-economic backgrounds to apply to higher status HE institutions. It is important to note for future evaluations of such programmes, therefore, that a comparison of participating vs. not participating schools would likely not provide a well-controlled study. Contrary to expectations, area of domicile did not have a significant influence of UCAS acceptance in our results. However, this was most likely a consequence of our sample only including participants from areas with the lowest participation in HE.
4.1. Limitations
Of course, given the study design, we cannot say with any certainty that the UniConnect intervention was the cause of this beneficial outcome. Although all learners were eligible to participate with UniConnect, in practice, any individual's opportunity to engage emerged from an unknown combination self-selection, school-selection and UniConnect-selection. Self-selection, because those who were uninterested in HE would be unlikely to engage. School selection, because each school had limited access to UniConnect interventions and may have selected learners deemed more likely to be succeed; and UniConnect selection because the resources allocated to different schools varied by location (rural vs. urban) and the number of pupils resident in target wards. The result is that those learners who did not engage at all were a heterogeneous group that did not engage for a variety of reasons.
Despite this limitation, the finding that there was a relationship between the extent of engagement and UCAS success provides better evidence for the efficacy of UniConnect interventions. Similarly, the apparent difference seen in the efficacy of the various interventions (some of which, like tutoring, seem to have provided little benefit despite the relatively large investment of time), indicates that the benefits of engaging with UniConnect are unlikely to have been solely due to learner selection. Other evidence comes from the overall UCAS success rate which was much higher than would be expected based on the POLAR3 quintile of this sample and above the overall average of 49% of UK students who took mainly Level 3 qualifications progressing to HE (Level 4 and above) in the year after they finished 16 to 18 study [33].
Unsurprisingly, prior attainment is considered a key factor in progression to HE. Indeed, it has also been shown that much (but not all) of the gap in socioeconomic differences in progression rates to HE can be attributed to socioeconomic differences in attainment [32]. The causal direction of this association, however, is a matter of some debate, with some proposing that lower attainment may be a result of perceived barriers to HE [32]. A second limitation of our study was therefore that no data were available on prior attainment.
Our data came from the schools the pupils attended, rather than the pupils themselves, eliminating a potentially difficult source of response bias. However, these schools would have garnered this information mainly from UCAS acceptances and pupil reports, rather than actual HE enrolment. A few students each year will accept a place but fail to enrol making UCAS acceptance only a proxy measure. Furthermore, in terms of the recording of outreach interventions, only outreach by consortium partners was recorded, and not attendance at events provided by other higher education providers outside of the region. It is therefore possible that participants had a higher engagement in activities than those recorded here.
4.2. Conclusions and future directions
Importantly, we have provided a robust statistical analysis showing that the UniConnect programme has been successful in its aim to help close the participation gap - with around 183 extra students in our sample progressing to HE than would be expected with no engagement in the programme. Our findings lead to clear recommendations for future research and practice in this area. First, in order to make best use of funding resources and pupil time, future intervention programmes should encourage pupils to participate in at least one - but no more than six - activities and should also consider the combinations of interventions shown to be most effective. Whilst combinations involving summer schools did seem to be effective, a combination of information, campus visits and master classes was also shown to be highly effective and would likely be more cost-efficient. Second, future evaluations of intervention programmes should exercise considerable caution before employing school-based comparison groups, because of the already evident differences between schools’ success in achieving pupil progression to HE.
Declarations
Author contribution statement
Adrian Burgess: Analyzed and interpreted the data; Wrote the paper.
Matthew Horton: Conceived and designed the experiments; Performed the experiments; Wrote the paper.
Elisabeth Moores: Conceived and designed the experiments; Wrote the paper.
Funding statement
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Data availability statement
The data that has been used is confidential.
Declaration of interests statement
The authors declare the following conflict of interests: The statistical analyses for this research and an internal report were commissioned to Adrian Burgess and Elisabeth Moores by Aimhigher West Midlands. Matthew S. Horton was employed by Aimhigher West Midlands at the time of writing the paper. Aston University and University of Birmingham both benefit from Aimhigher West Midlands funding. All authors were permitted full academic freedom to present their findings accurately and without bias.
Additional information
No additional information is available for this paper.
Acknowledgements
We are grateful to Gary Hilton for supporting access to the data in this report. We would also like to thank the UniConnect partnership, schools and the UniConnect evaluation working group for supporting this study. The statistical analyses for this research and an internal report were commissioned to AB and EM by Aimhigher West Midlands. EM also acknowledges support from the Centre for Innovation in Learning and Education (CILE), a Catalyst OfS funded project. The joint virtual centre aims to develop new knowledge in innovative education, business-engaged educational design and innovative delivery modes in undergraduate provision within UK Higher Education.
References
- 1.Education Policy Institute . 2018. Key Drivers of the Disadvantage gap. Literature Review.https://www.basw.co.uk/system/files/resources/EPI-Annual-Report-2018-Lit-review.pdf Education in England: Annual Report. From: [Google Scholar]
- 2.Emmons WR, Kent AH, Ricketts LR. The Demographics of Wealth: How Education, Race and Birth Year Shape Financial Outcomes. Accessed 29.04.2021 From: HFS_essay_1-2018.pdf (stlouisfed.org)
- 3.Eurostat: Statistics Explained Earnings Statistics - Statistics Explained (europa.Eu) https://ec.europa.eu/eurostat/statistics-explained/index.php/Earnings_statistics#Higher_level_of_education_yields_higher_earnings From:
- 4.Statistics Canada: Education, Training and Learning Statistics. https://www.statcan.gc.ca/eng/subjects-start/education_training_and_learning From:
- 5.James R., Bexley E., Anderson A., Devlin M., Garnett R., Marginson S., Maxwell L. 2008. Participation and Equity : a Review of the Participation in Higher Education of People from Low Socioeconomic Backgrounds and Indigenous People Centre for the Study of Higher Education, Melbourne, Vic.https://melbourne-cshe.unimelb.edu.au/research/archived-research/participation-and-equity From: [Google Scholar]
- 6.Skilbeck M. The Higher Education Authority; Dublin: 2000. Access and Equity in Higher Education: an International Perspective on Issues and Strategies. From: Access and Equity in Higher Education: An International Perspective on Issues and Strategies (edepositireland.ie) [Google Scholar]
- 7.UNICEF Office of Research . 2018. An Unfair Start: Inequalities in Children’s Education in Rich Countries.https://www.unicef.org/reports/unfair-start Innocenti Report Card 15. Innocenti, Florence. From: [Google Scholar]
- 8.Joseph Rowntree Foundation UK Poverty 2018: A Comprehensive Analysis of Poverty Trends and Figures. https://www.jrf.org.uk/report/uk-poverty-2018?gclid=EAIaIQobChMIwOSjhDX4gIVQ7DtCh0GJgmvEAAYASAAEgKGwPD_BwE From:
- 9.Secretary of State for Education Access and Participation: Secretary of State for Education Guidance to the Office for Students (OfS) https://www.officeforstudents.org.uk/media/1112/access-and-participation-guidance.pdf From:
- 10.Gorard S., Smith E., May H., Thomas L., Adnett N., Slack K. Higher Education Funding Council for England (HEFCE); Bristol: 2006. Review of Widening Participation Research: Addressing the Barriers to Participation in Higher Education.https://dera.ioe.ac.uk/6204/1/barriers.pdf From: [Google Scholar]
- 11.Doyle M., Griffin M. Raised aspirations and attainment? A review of the impact of Aimhigher (2004–2011) on widening participation in higher education in England. London Rev Educ. 2012;10:75–88. [Google Scholar]
- 12.Gorard S., Smith E. Beyond the ‘learning society’: what have we learnt from widening participation research? Int J Lifelong Educ. 2006;25:575–594. [Google Scholar]
- 13.Riddell S., Edward S., Boeren E., Weedon E. 2013. Widening Access to Higher Education: Does Anyone Know what Works? A Report to Universities Scotland.http://www.universities-scotland.ac.uk/uploads/WideningAccessToHE-CREID.pdf From: [Google Scholar]
- 14.Younger K., Gascoine L., Menzies V., Torgerson C. A systematic review of evidence on the effectiveness of interventions and strategies for widening participation in higher education. J Furth High Educ. 2019;43:742–773. [Google Scholar]
- 15.Gorard S., See B.H., Davies S. 2012. The Impact of Attitudes and Aspirations on Educational Attainment and Participation.http://www.jrf.org.uk/sites/files/jrf/education-young-people-parents-full.pdf From: [Google Scholar]
- 16.Bergin D.A., Cooks H.C., Bergin C.C. Effects of a college access program for youth underrepresented in higher education: a randomized experiment. Res High Educ. 2007;48:727–750. [Google Scholar]
- 17.Robinson D., Salvestrini V. 2020. The Impact of Interventions for Widening Access to Higher Education: a Review of the evidence.https://taso.org.uk/wp-content/uploads/Widening_participation-review_EPI-TASO_2020.pdf Report to TASO: Transforming Access and Student Outcomes in Higher Education. From: [Google Scholar]
- 18.Bowman N.A., Kim S., Ingleby L., Ford D.C., Sibaouih C. Improving college access at low-income high schools? The impact of GEAR UP Iowa on postsecondary enrollment and persistence. Educ Eval Policy Anal. 2018;40:399–419. [Google Scholar]
- 19.Emmerson C., Frayne C., McNally S., Silva O. Department for Education and Skills; 2006. Aim Higher: Excellence Challenge: A Policy Evaluation Using the Labour Force Survey.https://www.ifs.org.uk/publications/3801 Research Report RR813. From: [Google Scholar]
- 20.Morris M., Golden M. Department for Education and Skills; 2005. Evaluation of Aimhigher: Excellence challenge Interim Report.https://nfer.ac.uk/publications/EIC04/EIC04.pdf Research Report RR648. London: DfES. From: [Google Scholar]
- 21.Chilosi D., Noble M., Broadhead P., Wilkinson M. Measuring the effect of Aimhigher on schooling attainment and higher education applications and entries. J Furth High Educ. 2010;34:1–10. [Google Scholar]
- 22.Horton M., Hilton G. 2019. Has Engagement in AHWM NCOP Increased Students’ Higher Education Progression Rates?https://aimhigherwm.ac.uk/research-impact/research-impact-he-progression/ Interim NCOP Research Report. From: [Google Scholar]
- 23.HEFCE . 2012. POLAR 3: Young Participation Rates in Higher Education.https://dera.ioe.ac.uk/15706/7/POLAR3_Redacted.pdf From: [Google Scholar]
- 24.English Indices of Deprivation. 2019. https://www.gov.uk/guidance/english-indices-of-deprivation-2019-mapping-resources From: [Google Scholar]
- 25.Income deprivation affecting children index. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/835115/IoD2019_Statistical_Release.pdf From:
- 26.Heberle H., Meirelles G.V., da Silva F.R., Telles G.P., Minghim R. InteractiVenn: a web-based tool for the analysis of sets through Venn diagrams. BMC Bioinform. 2015;16:169. doi: 10.1186/s12859-015-0611-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Heck R.H., Thomas S.L., Tabata L. Routledge; 2012. Multilevel Modeling of Categorical Outcomes Using IBM SPSS. [Google Scholar]
- 28.Department for Education . 2017/18. Destinations of Key Stage 4 and 16-18 Students, England.https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/860135/Destinations_main_text_2020_REV.pdf From: [Google Scholar]
- 29.McCaig C., Stevens A., Bowers-Brown T. Higher Education Research Network, Sheffield. 2006. Does Aimhigher work? Evidence from the national evaluation; pp. 1–16.https://shura.shu.ac.uk/2382/1/Does_Aimhigher_work_CM_AS_TB_2007.pdf From: [Google Scholar]
- 30.Ireland E., Golden S., Morris M. Department for Education and Skills; 2006. Evaluation of Integrated Aimhigher: Tracking Surveys of Aimhigher.https://dera.ioe.ac.uk/6562/1/RR811.pdf Research Report RR811. London: DfES. From: [Google Scholar]
- 31.Hatt S., Baxter A., Tate J. It was definitely a turning point! A review of Aimhigher summer schools in the south west of England. J Furth High Educ. 2009;33:333–346. [Google Scholar]
- 32.Chowdry H., Crawford C., Dearden L., Goodman A., Vignoles A. Widening participation in higher education: analysis using linked administrative data. J R Statist Soc A. 2013;176:431–457. [Google Scholar]
- 33.HESA . 2019. Who's studying in HE?https://www.hesa.ac.uk/data-and-analysis/students/whos-in-he From: [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The data that has been used is confidential.