How you teach and who you teach both matter: lessons from learning analytics data

ABSTRACT To explore potential effects of disadvantage on engagement and attainment under different teaching and assessment regimes, the influence of pedagogic changes implemented during the COVID-19 pandemic on attainment and engagement of students from different backgrounds were compared using a cohort-study design. Learner analytics and attainment data from first year undergraduate students during three learning regimes were compared: (i) in-person teaching and assessment, (ii) in-person teaching with online assessment, and (iii) online teaching and assessment. The gap in end-of-year mark between disadvantaged students and their peers was widest when teaching and assessment was online, with poorer outcomes for disadvantaged students, although the gap in the percentage of students passing all their modules did not change. Overall, online teaching and assessment during the pandemic was associated with a widening attainment gap between disadvantaged students and their peers. Possible explanations for this are discussed, including the relationship between attainment and engagement. Higher Education providers should monitor and review the potential implications of their chosen education strategy on different groups of students: how you teach and who you teach are both important.


Introduction
The COVID-19 pandemic forced many higher education providers to move much of their teaching and assessment online in 2020 and 2021, giving rise to serious concerns about the potential effects of digital poverty on education (i.e.exclusion from education due to lack of appropriate hardware, software, or internet connectivity).Using survey results, Rodríguez-Planas (2022) reported that retention rates fellup to 34% of students considered dropping a courseand this rate was even higher in low-income students (see also Bird, Castleman, and Lohner 2022;Kofoed et al. 2021).In contrast, El Said (2021) reported no negative effects, and that the shift to online learning could be beneficial in terms of cost-effective expansion of provision.Similarly, Kizilcec, Makridis, and Sadowski (2021) reported higher engagement during the pandemic amongst existing online students from lower income regions.As the ongoing risks of COVID-19 appear to be reducing, higher education providers are considering what the 'new normal' should entail, commonly trying to harness some of the perceived advantages and flexibility of online learning, whilst still providing aspects of an on-campus experience.Ewing (2021) reported an international study of vice-chancellors' views of COVID-19's impact on higher education and what the future might hold; some reported that COVID-19 has 'legitimised' online learning and most supported some kind of 'blended learning' in the future which involves aspects of online and on-campus teaching (see also Saha et al. 2022;Xie, Siau, and Nah 2020).
A UK survey of 33,726 students suggested that 45% wanted a mix of on campus and online learning, 42% wanted only on-campus provision, and the remainder (13%) wanted online learning (Jisc 2022).Clearly, some element of online learning is wanted by many universities and students, but the potential effects of such a pedagogical shift on student engagement, learning and attainment have not yet been fully elucidated.There is some evidence that blended learning can be highly effective (e.g.Bernard et al. 2014;Spanjers et al. 2015;Strelan, Osborn, and Palmer 2020) and can increase engagement and learning (e.g.Akçayir and Akçayir 2018;Haugom 2022;Northey et al. 2015;Snowball 2014), although that success may depend on the teaching methods employed (e.g.Foo, Cheung, and Chu 2021;Lin, Huang, and Lin 2022;Orlov et al. 2021) or on the student (e.g.Cacault et al. 2021;DePaola, Gioia, and Scoppa 2022).
There is a strong relationship between student engagement and attainment, even if that relationship is not considered causal (see e.g.Credé, Roch, and Kieszczynka 2010 for a review, or the metaanalysis by Lei, Cui, and Zhou 2018), and the same relationship appears to hold with online teaching.For example, Virtual Learning Environment (VLE) activity and attainment have been shown to correlate for both in-person and online courses (Agudo-Peregrina et al. 2014;Macfadyen and Dawson 2010;Mogus, Djurdjevic, and Suvak 2012;Morris, Finnegan, and Wu 2005;Ramos and Yudko 2008;Summers, Higson, and Moores 2021;Waheed et al. 2020;You 2016).In a longitudinal study of 106 students Saqr et al. (2023) found that whilst high engagement over a long period and several time points across a four-year programme of study can predict high achievement, a short period of disengagement can predict lower achievement in students with low engagement.Peach et al. (2019) similarly showed that whereas high performers showed 'diverse temporal engagement', low performers were more likely to fall in a massed learning or 'crammers' cluster, illustrating that the consistency as well as the amount of engagement is important.Many higher education providers now use learning analytics systems to track students' engagement with their courses, both online and in-person.One consequence of this is that universities have an opportunity to use these data to conduct interventions that support attainment and progression for students based on their engagement, rather than their demographics (Foster and Siddle 2020;Summers, Higson, and Moores 2021).This could enable institutions to use such systems to improve equality of outcomes between different student groups (Khalil, Slade, and Prinsloo 2023;Williamson and Kizilcec 2021).The use of learning analytics data to target such interventions has the advantage of avoiding stigmatisation of students with particular characteristics, as well as increasing the efficiency of the learning process.Celik et al. (2023) systematically reviewed publications on how learning analytics have been used during the pandemic, providing insight on usage, challenges, and potential benefits of this technology.
The impact of the COVID-19 pandemic on teaching in Higher Education has provided a useful 'natural experiment' to help evaluate the impact of different teaching methods.Orlov et al. (2021) conducted a study of economics students results pre-and peri-pandemic across four different institutions and argued that it was not 'who you teach', but 'how you teach' that mattered.Student characteristics showed little association with a general decline observed in student performance during the pandemic (see also Lin, Huang, and Lin 2022).Summers, Higson, and Moores (2023) provided an analysis of learning analytics data on engagement for different groups of students and found that students from disadvantaged backgrounds showed differences in digital engagement pre-vs.peri-pandemic, on average watching fewer recorded lectures peri-pandemic than they had previously; this might be expected to affect attainment, although this was not explicitly tested.Altindag, Feliz, and Tekin (2021) reported an increase in average grades following the inperson teaching/ online assessment regime, but more so for students living in areas with better broadband technology, suggesting that access to technology may play a role in student attainment, at least when assessment is online.
The time before and during the pandemic have provided a unique opportunity to compare the impact of different teaching and assessment approaches on students from disadvantaged and nondisadvantaged backgrounds.Using measures of attainment and engagement from a large and diverse sample the data available to us allowed further investigation of Orlov et al.'s contention that who you teach does not matter.This work expanded on Summers, Higson and Moores' (2023) findings of different engagement patterns amongst disadvantaged students pre-and peripandemic, to include examination of the relationship between engagement and attainment; Summers, Higson, and Moores (2023) did not analyse attainment data.Specifically, the effect of three teaching and assessment regimes on student attainment was investigated for both disadvantaged (IMD [Index of multiple deprivation] quintiles 1-2) and non-disadvantaged (IMD quintiles 3-5) students: (i) in-person teaching with in-person assessment, (ii) in-person teaching with online assessment and (iii) online teaching with online assessment.We also explored the extent to which differences in attainment were associated with different patterns of engagement and how this interacted with the mode of teaching and assessment.

Sample participants
Aston University is a medium-sized UK university of around 11,000 students which has a student body that is more diverse, in terms of both ethnicity and socio-economic background, than most other UK universities.For this study, the records of three cohorts of first year full-time home undergraduate students who began their studies in 2018/19, 2019/20 and 2020/21 were analysed (initial sample of 5,182 students).First year undergraduates were used because the learning analytics system data was only available across all three years of interest for the first-year cohorts.
Students who were repeating a year, did not complete their first year (2018/19 and 2019/20 cohorts) or were not listed as current at the end of the academic year in June 2021 (2020/21 cohort) were removed from the sample.Additionally, and to enable normalisation of learning analytics data (see below), students were omitted from the sample if their course had not been monitored by the learning analytics system in each of the three years, as were students on courses with fewer than ten members.For the remaining students their home postcode was matched to a UK-wide adjusted index of multiple deprivation (IMD) quintile.IMD is a readily available measure of disadvantage which has a moderate correlation with permanent income deprivation (Jerrim 2021) and is therefore likely to be related to digital poverty, although this is a complex and arguably 'non-binary' issue (see e.g.Hernandez and Faith 2022).IMD quintiles are not normally comparable between the countries of the UK, but Abel, Barclay, and Payne (2016) derived an adjustment such that indices from three of the constituent countries of the UK can be compared with each other.This adjustment has been updated for the most recent 2020 indices by Parsons (2021) and was used here; namely that indices for Scotland, Wales and Northern Ireland were adjusted to be comparable to those from England.After removing students whose IMD quintile could not be identified, due to unrecognised postcodes, a sample of 4,582 students remained (see Table 1).The UK Office for Students (OfS) considers students from IMD quintiles 1 and 2 as meeting widening participation criteria (disadvantaged), whilst students from quintiles 3-5 are considered non-disadvantaged; students were divided into two categories that aligned with this distinction.There were 29 courses amongst the sample of students covering a wide range of the courses offered, including Business, Engineering, English Language and Literature, Computer Science, Law, Mathematics, Optometry, Pharmacy and Psychology.Some courses, such as Optometry and Pharmacy, required limited on-campus attendance during online teaching to participate in laboratory classes.

Sample data
For each student, two items of attainment data were obtained; end of year mark and credits obtained.The end of year mark (%) is the average mark of all first-year modules that the student had completed.Each credit corresponds to ten hours of study, and a student is considered to have completed a stage of study (year) if they obtain 120 credits.A student can only reach 120 credits if they pass all their modules (score at least 40%) and can only progress to the following stage of study if they do so.
Engagement data from the university's learning analytics system were also analysed.All undergraduate modules are managed through a Virtual Learning Environment (VLE), where announcements, timetables, online synchronously delivered lectures, and course materials can be accessed.Since 2018, in-person attendance at lectures and seminars has been electronically recorded by students swiping their identity card, though neither attendance nor the act of recording attendance is compulsory for home students.Additionally, all lectures are recorded and available through the VLE via a lecture capture system (LCS).The learning analytics system aggregates the log data from the VLE, attendance recording system and lecture recordings on a daily basis.Following Summers, Higson, and Moores (2023), we used the data from three sources: (i) VLE course accesses: number of times the student accessed course materials, (ii) Attendance: total number of in-person classes and online classes that the student 'attended'note that this could encompass either in-person attendance or virtual attendance, but the presentation was 'live' and 'synchronously' delivered in each case, and (iii) LCS: number of times the student viewed recorded lectures.A fourth data feed, number of library books checked out, was dropped from this analysis as library use was so low in 2020/21 that the median and modal number of books checked out was zero.

Analyses
All the statistical analyses were computed using R 4.2.0 (R Core Team 2022).Linear-and Logistic mixed-effects models were computed using lmer and glmer, respectively, from the package lme4 (Bates et al. 2015).The significance of the effects of the main factors of the linear mixed-effects models were evaluated using the package lmerTest (Kuznetsova, Brockhoff, and Christensen 2017) which implemented the Satterthwaite approximation to estimate the denominator degrees of freedom of the F statistic.Estimated marginal means of the linear models were computed using emmeans (Lenth 2021).
The relationship between end of year mark, teaching mode and IMD was explored with the following linear mixed-effects model: where Mark i is the mark of the ith student, b 0 is the intercept, b x are the coefficients, Mode is the teaching mode which could be Teaching: In-person; Assessment: In-person, Teaching: In-person; Assessment: Online, or Teaching: Online; Assessment: Online, and IMD is either Q12 (disadvantaged) or Q345 (non-disadvantaged).The controls are Sex (Female or Male) and Ethnicity (White, Asian, Black, Other or Unknown). 1 The term b i is a zero-mean normally distributed parameter to account for any differences in mean grade between courses for each teaching/assessment mode.The term e is the residual error.Note, the anova function was used to compare this model with two others and it was found to be the best fitting model by minimum Akaike information criterion (Akaike 1974); the other models considered had the random-effects structure of Course nested by IMD, and where the random-effects structure was just a simple random intercepts model for Course.
Nesting by students did not occur since students only appeared in a single academic year (repeating students were excluded from analyses).We also examined the effects of mode of teaching and assessment mode and IMD on pass (120 credits obtained) and fail (<120 credits obtained) with the following logistic model: where P(credits = 120) and P(credits < 120) are the probabilities of students obtaining 120 credits, or not, respectively.We used this split of either achieving 120 credits or not, because all students who do not achieve 120 credits must do some form of extra work (either essays or resitting examinations) in order to continue their studies in the next year or stage, although we accept that those students with less than 120 credits may be a highly heterogenous group.

Clustering of engagement data
Cluster analysis (kmeans clustering in base R) was used to identify groups of students who were similar in terms of their learning analytics data.Prior to clustering, for each student, the average weekly data for the three sources of learning analytics data was computed for the 21 teaching weeks of the 2018/19, 2019/20 and 2020/21 academic years.For the 2018/19 and 2019/20 academic years, synchronous or live teaching was conducted entirely on campus, whereas for 2020/21 teaching was conducted almost entirely online, either synchronously (live) or asynchronously (e.g.pre-recorded lectures).For each student the weekly data -VLE course accesses, Attendance, LCS was then normalised to z-scores by course and cohort; i.e. for a given data source in each course and in each year the mean and standard deviation were computed and each students' weekly data was normalised by subtracting the relevant mean and dividing by the relevant standard deviation; note that this normalised data was used only to generate the clusters by making the data comparable across all year groups, all other analyses were carried out on the raw data.
The optimum number of clusters for the whole dataset was assessed using gap analysis with clusGap from the cluster library (Maechler et al. 2022).The gap statistic is a measure of the intracluster sum of squares for a given clusteringits compactnesswhich can be compared with that from a reference dataset with no clustering.The clusGap function was used to generate reference datasets by bootstrapping (n = 500).Gap statistics were computed for up to six clusters and the globalSEMax rule (Dudoit and Fridlyand 2002) demonstrated that the maximum gap statistic occurred for three clusters (see Figure S1).For each cluster, the relationship between attainment (end of year mark), IMD and mode was explored using the linear model defined above.

Average end of year marks
There was a significant effect of teaching mode on attainment [F (2, 25.7) = 3.887, p = .033]with attainment being highest in the cohort for which teaching was in-person and assessment was online.
There was also a significant attainment gap [main effect, IMD, F (1, 4525.1)= 17.456, p < .001];disadvantaged students achieved lower marks than their non-disadvantaged peers.A significant interaction between teaching mode and IMD [F (2, 4518.3)= 4.145, p = .016]showed that this difference was largest in the online teaching and assessment cohort (Figure 1).Post-hoc comparisons indicated no significant attainment gap between disadvantaged and non-disadvantaged students when teaching and assessment were in-person (0.6% pts [t (4497.6)= 1.145, p = .252])or when teaching was in-person and assessment online (0.8% pts [t (4518.0)= 1.509, p = .131]).Only when teaching and assessment were both online, was there a significant attainment gap (2.6% pts [t (4514.4)= 4.893, p < .001]).To put this in context, an attainment gap of 2.6% pts is equivalent to a quarter of a degree classification boundary and, if the gap were eliminated, would result in an uplift of the classification for 22% of disadvantaged students when teaching and assessment were online.It should be noted that the interaction is driven by both the non-disadvantaged students achieving higher marks in the online assessment regimes, as well as by the disadvantaged students achieving lower marks.Both Sex [F (1, 4431.2) = 70.978,p < .001]and Ethnicity [F (4, 4526.8)= 24.990,p < .001]were also significantly associated with attainment 2 , with white students and female students obtaining higher marks.These are well-known attainment gaps for both characteristics (e.g.HEFCE 2018).

Credits obtained
The estimated probabilities from the logistic model of students passing the 1st year by obtaining 120 credits at the first attempt are plotted in Figure 2. The coefficients of the logistic model are provided in Table 2. To aid interpretation of the logistic model and determine the effects of IMD and teaching mode on credits, pairwise comparisons between disadvantaged and non-disadvantaged students were calculated for each teaching mode (see Table S1); only during online teaching and assessment was the log odds of obtaining 120 credits in disadvantaged students lower than that of their counterparts (log odds = 0.412, Z = 3.296, p < .001).For the in-person teaching and assessment cohort and the in-person teaching/ online assessment cohort the differences were not significant (log odds 0.226, Z = 1.641, p = .101and log odds 0.210, Z = 1.565, p = .118respectively).The change in log odds between different teaching modes for disadvantaged and non-disadvantaged students are in Table 3.Only one comparison was significant, indicating a significant decrease in log odds of obtaining 120 credits for disadvantaged students in fully online vs. fully in-person teaching and assessment.To determine if there was an interaction between Mode and IMD we compared the difference in log odds between disadvantaged and non-disadvantaged students for the three different contrasts in teaching mode.None of these contrasts were significant (see Table S2), indicating there was no interaction between IMD and mode.Coefficients are the bs from Equation ( 2); e.g. the line 'T:Online/A:Online' corresponds with b 1 when teaching and assessment are both online.The intercept is the estimate of the log odds for Disadvantaged students when Teaching and Assessment were inperson (b 0 in equation ( 2)).Other coefficients are changes in the log odds with reference to Disadvantaged students and Mode = T:In person/A:In person.
As Sex and Ethnicity were simple covariates in the logistic model (see equation ( 2)) interpretation of their effects can be read directly from Table 2. Male students were significantly less like likely to achieve 120 credits than female students, and Black students were significantly less likely to achieve 120 credits than white students. 3

Engagement and attainment during the different teaching modes
The cluster analysis used to identify groups of students who were similar in terms of their learning analytics data revealed three clusters that can be broadly defined as: (1) High engagers: Students who had high attendance of synchronously delivered teaching (online synchronous and/or in-person), LCS views, and VLE usage, (2) Synchronous-teaching engagers: Students with high attendance of synchronously delivered teaching (online synchronous and/or in-person), but low LCS views, and average VLE usage, and (3) Low engagers: Students whose engagement was low for attendance of synchronously delivered teaching (online synchronous and/or in-person), LCS views, and VLE usage.
Summary engagement data for each of the three clusters is in Table 4 which reveals how interaction with the LCS and VLE increased as teaching moved online during the pandemic and, due to a reduction in the number of synchronous teaching sessions (whether in-person or online), attendance decreased.The differences in engagement between the clusters are also apparent, e.g.synchronousteaching engagers and high engagers have similar attendance profiles but different LCS views and VLE engagement.Estimate is the modelled difference in log odds between obtaining 120 credits or not for the stated contrast.Overall, about 20% of students were 'high engagers', 38% were 'synchronous-teaching engagers' (in-person and/or online) and 42% were 'low engagers', but there was variation in the proportion of disadvantaged and non-disadvantaged students in the different clusters across teaching/assessment modes, in particular with fewer disadvantaged students in the low engager cluster during in-person teaching and assessment (see Figure 3).To explore how the relationship between IMD and engagement varied with teaching mode, a series of three chi-squared analyses were performed.Only for the in-person teaching and assessment regime were engagement cluster and IMD significantly related (χ ( 2) 2 = 15.594,p < .001),with disadvantaged students being more likely to be 'high engagers' and less likely to be 'low engagers', whilst the proportion of 'synchronous-teaching engagers' was similar for both groups.When teaching moved online, the proportion of students in the 'synchronous-teaching engagers' cluster reduced substantially for both groups (on average between 9 and 13% pts for non-disadvantaged and disadvantaged students respectively).Neither the online teaching and online assessment teaching regime nor the in-person teaching with online assessment regime showed significant associations between cluster and IMD (χ ( 2) 2 = 1.693 and χ (2) 2 = 2.757 respectively).In other words, there was a more comparable pattern of engagement between these groups under both online conditions.
We investigated whether attainment in each of the different teaching regimes was affected by engagement and disadvantage in a similar way to that found for all students (Figure 1). Figure 4  (A-C) shows the model-estimate end of year marks achieved by each IMD group under the three different teaching regimes and by engagement cluster.Figure 4(A-C) shows generally lower attainment for disadvantaged students within each cluster and each teaching regime, but most notably amongst the low engagers during online teaching and assessment.We ran three linear mixedeffects models, one for each teaching mode (see Table 5).The attainment gap between disadvantaged students and their peers was greatest in the low engagement cluster and when teaching was online (4.5% pts, see Figure 4(C)).IMD was a significant factor when teaching was in-person and assessment was online (F (1,1473.6)= 5.105, p = .024),and when teaching and assessment were both online (F (1,1627.1)= 14.288, p < .001),but not when teaching and assessment were both in-person (F (1,1397.8)= 2.722, p = .099).Only when teaching and assessment were online was there a significant interaction between cluster (engagement) and IMD (F (2,1620.8)= 3.970, p = .019),with pairwise comparisons indicating the largest attainment gap in the low engager cluster (see flagged differences in Figure 4 and Table S3).That is, the main driver of the initial two-way interaction found between IMD and Teaching Mode for all students (Figure 1) was the attainment gap in the low engagers cluster in online teaching and assessment.During fully online teaching and assessment the nondisadvantaged low engagers appeared to maintain the benefits of online assessment shown by other groups, whilst the disadvantaged low engagers showed a decline; a cascade of disadvantage.

Discussion
In summary, our results showed that students from disadvantaged backgrounds (as defined above) fared worse than their peers during online teaching and assessment, both in terms of obtaining full  credits and marks obtained (see also Bird, Castleman, and Lohner 2022;Kofoed et al. 2021).Whilst disadvantaged students were more likely than their peers to be in the high engagement cluster for inperson teaching and assessment, the association between disadvantage and engagement disappeared thereafter.However, despite more similar patterns of engagement in the two groups, the attainment gap was larger during online teaching and assessment, particularly for the group in the low engagement cluster.Disadvantaged students during the period of online teaching and assessment therefore showed a 'triple whammy' of effects; a greater likelihood to be in the lower engagement cluster than previously (in turn associated with lower attainment), lower overall attainment associated with their background, and lower attainment associated with the online teaching and assessment regime (at least in comparison to teaching in person/ assessment online conditions).
Our findings could be interpreted as a sign of digital poverty and, for some students, this may have been the case.Altindag, Feliz, and Tekin (2021) found a weaker increase in grades during in person teaching/ online assessment for those students living in areas with inferior broadband technology.Fifty-four percent of the disadvantaged students, however, remained in the high-engagers or synchronous-teaching engagers clusters during online teaching and assessment; clusters for which attainment remained high.A large proportion of students from both groups fell into the low engagement cluster, although it was for the disadvantaged students that this proportion increased most during online teaching, albeit only to a level similar to the non-disadvantaged group.Thus, the pattern of engagement alone cannot explain the attainment gap between these groups during online teaching and assessment.
Our results are in contrast with those of Orlov et al. (2021) which suggested that who you teach does not matter, and with El Said's (2021) results that found no negative effects of online learning, but align well with findings from several other studies (e.g.Bird, Castleman, and Lohner 2022;Kofoed et al. 2021;Rodríguez-Planas 2022;Summers, Higson, and Moores 2023) that reported some negative effects of online learning, particularly for students already disadvantaged in other ways.Of course, there are many things which changed during the pandemic unrelated to teaching and these may have also affected both engagement and attainment.For example, illness and financial insecurity were known to have affected disadvantaged groups more than their peers (see e.g.Rodríguez-Planas 2022) and studying at home is also likely to be more difficult for disadvantaged groups, whether or not there is a lockdown or appropriate digital infrastructure available (see, e.g.Bashir et al. 2021).Fabian et al. (2022, 5) coined the term e-learning capital as 'a measure of self-expressed ability and resources to utilise the online learning environment' and found that it was positively correlated with study skills engagement during the pivot to online learning; whether students from disadvantaged backgrounds have lower e-learning capital and would thus be less likely to thrive during online teaching remains an open question.
Alternatively, the widening attainment gap during online teaching could be influenced by cohort differences.Students seeking entry to university did not sit examinations in summer 2020, meaning thatfor the majority of studentsteacher assessed grades (TAGs) were used for admissions purposes, rather than examinations nationally marked and moderated.In addition, much of the 2020/1 cohort would have been without their usual schooling (teaching or revision) for approximately six and a half months prior to university enrolment.
Although we identified three distinct patterns of student engagement, there was no cluster that showed lower engagement with synchronous-teaching and high digital engagement (LCS views and VLE accesses).Anecdotally, many students noted the benefits of synchronous sessions being online, specifically including the ability to anonymously ask questions in the 'chat' function of the software used (see also Hollister et al. 2022).There may be an advantage of attending synchronous teaching over pre-recorded content (see Fabriz, Mendzheritskaya, and Stehle 2021).Our division of participants into three clusters cannot be considered definitive.Although we used a well-established and widely used method of clustering, based on the best available criteria, cluster analysis can only ever be considered as exploratory, and variations in the agglomeration method used, or the variables included, may have produced somewhat different classifications.However, the clusters produced were intuitively plausible and provided understandable results.
Our findings build on those of Summers, Higson, and Moores (2023) that found changes in average engagement of disadvantaged students peri-pandemic but did not include any analysis of attainment in terms of either marks or credits obtained.In addition, analyses using clusters in the present study were able to show how students in each group moved between clusters during the different teaching regimes, precluding the possibility that average measures of engagement were affected by only a few students.In the present study, we also showed that the divergence in grade attainment between advantaged and disadvantaged students during the online teaching and assessment regime was driven principally by students in the low engagement cluster, further potential evidence of digital poverty and of the relationship between engagement and attainment.
The most significant limitation of this study is that the pandemic had profound effects on peoples' lives that extended well beyond teaching, and these may have been differentially affected different groups of students.It is also possible that the effects of enforced online teaching during a pandemic may not generalise to times when students have chosen online learning as their preferred medium of study (see e.g.Kizilcec, Makridis, and Sadowski 2021 for contrasting results).Similarly, these effects, found when staff had to hastily adapt to online teaching (see, e.g.Hodges et al. 2020;Orlov et al. 2021), may not apply in post-pandemic times now that staff are much more experienced in teaching in this format.
There are also some limitations related to the quality of the learning analytics data used in this study.For example, the learning analytics system measure of digital engagement only counted individual login events and did not capture other factors that may have may influenced students' experience, such as what type of device they are using (e.g.mobile phone, tablet, or PC), or whether the internet connection allowed them to watch, listen or contribute fully; the effects of digital poverty may therefore go beyond simply the ability to log in and be 'non-binary' (see Hernandez and Faith 2022).
Additionally, our analyses did not investigate any temporal aspects of engagement; there is increasing evidence that consistency of engagement as well as the total amount of engagement may be important (see e.g.Peach et al. 2019;Saqr et al. 2023;Summers, Higson, and Moores 2021).There are always some doubts over the validity of learning analytics data as it is typically incomplete, and our data is no exception (see e.g.Selwyn 2019 for a discussion).To overcome some of the potential issues of incomplete data, our data were normalised before clustering, to ensure that the student groups that emerged were driven by relative engagement within each teaching regimen and not by differences in the absolute levels of engagement between them.
Other limitations include the use of IMD as a measure of disadvantage.The IMD is an area-based measure that is an amalgamation of several indices of deprivation and, like all area-based measures IMD is subject to error in that i.e. non-deprived persons may live in disadvantaged areas and vice versa (Jerrim 2021), so the attainment gap due to IMD that we report here is likely to underestimate the real impact of deprivation.Similarly, as ethnicity was associated with IMD status in our sample, we may have underestimated the impact of disadvantage here by including ethnicity as a control variable.Analyses were conducted both with and without these controls and the attainment gap was indeed widerand significant for all teaching modeswithout them, this difference reflecting the influence of well-known ethnicity and gender attainment gaps.Finally, the results of this study are unlikely to generalise to countries that are less digitally developed (see, e.g.Adnan and Anwar 2020;Giatman, Siswati, and Basri 2020).
Although recent studies have reported on effects of the pandemic on engagement using learning analytics systems (e.g.Xu and Wilson 2021;Zhang, Taub, and Chen 2021), and on marks (e.g.Orlov et al. 2021), there has been a relative lack of research on equality issues using learning analytics (Williamson and Kizilcec 2021; but see Hlosta et al. 2021;Summers, Higson, and Moores 2023), despite the fact that learning analytics systems have valuable data to reveal insights about learners, learning, courses, and the university itself.Summers, Higson, and Moores (2023, 7) also found differential effects for asynchronously versus synchronously delivered digital material, suggesting 'a more nuanced, than a simplistic or all-encompassing view of "digital poverty"'.Overall, therefore, it seems reasonable to speculate that different delivery and assessment methods may differentially affect different groups of students because of factors includingbut not necessarily exclusive todigital poverty.In other words, recorded online lectures as a sole or main source of provision, seems to reduce engagement in students, particularly those from disadvantaged backgrounds.In turn, lower engagement is associated with lower attainment.For providers wanting to close attainment gaps, this has important implications for their broader education strategy; 'how you teach' and 'who you teach' both matter.
It would be advisable for Higher Education providers to consider (and monitor) the potential implications of their education strategy on different groups of students in terms of engagement and attainment as well as reported preferences (see also Celik et al. 2023).The broader context of on-campus experience may offer something that the online experience does not (see e.g.Buhl-Wiggers, Kjaergaard, and Munk 2023).Providers wishing to provide hybrid or online learning could endeavour to monitor engagement with a view to providing interventions to ensure good engagement by all students, or experiment with different formats of online delivery to ensure that students show good engagement with their studies in a way that allows them to fulfil their potential.There is increasing evidence that patterns of engagement and disengagementincluding at different times of the yearmay be at least as important as average engagement across the year (e.g.Saqr et al. 2023;Summers, Higson, and Moores 2021).Further research should attempt to elucidate the elements which make an on campus or a blended experience successful.

Figure 3 .
Figure 3. Relative proportions of students in each cluster by IMD (left panel = disadvantaged, right panel = non-disadvantaged) and (T)eaching/(A)ssessment mode (different bars).Numbers in brackets are the count of students in each cluster and IMD.

Figure 4 .
Figure 4.Estimated marginal means of the linear mixed-effects model for each (T)eaching/(A)ssessment mode (different panels), end-of-year mark (y axis), cluster (x axis) and IMD (different symbols).Error bars indicate 95% confidence interval of the mean.Asterisks indicate significant differences in attainment between disadvantaged and non-disadvantaged students for the relevant cluster (* p < .05;** p < .01;*** p < .001).

Table 1 .
Demographics for each of the three cohorts comprising the sample for each (T)eaching and (A)ssessment mode.

Table 3 .
Pairwise comparisons for the log odds of obtaining 120 credits between different (T)eaching/(A)ssessment modes for either disadvantaged students or for non-disadvantaged students.

Table 4 .
Cluster summary statistics: Number of students (n), and mean and standard deviation of the weekly levels of Attendance, LCS views and VLE Course Accesses broken down by cluster and (T)eaching/(A)ssessment mode.Note that clustering was performed on the complete dataset (normalised as described in the text) but is presented separately for each teaching mode.

Table 5 .
Results of ANOVA for three linear-mixed effects models, one for each teaching/assessment mode, relating end of year mark with Cluster and IMD.