Coherently Organized Digital Exercises and Expositions

We describe an organizing principle for online learning materials; we term coherently organized digital exercises and expositions. Larger in scale than individual lessons but smaller than a programme of study, this innovation in instructional practice is increasingly guiding our thinking in the development of university mathematics courses. Essentially we have taken the book and put it inside automatically assessed online quizzes. In doing this, we embrace the potential provided by new technology to implement evidence-based practices such as spaced retrieval practice. This paper discusses details of this innovation, and how we have implemented it. On the basis of these experiences, we believe this innovation has the potential to change the model of education for university mathematics courses in substantial and non-trivial ways.


INTRODUCTION
This paper considers technological advances which led to innovative instructional practices, and discusses their impact on learning mathematics. In particular, in this short theoretical report, we describe an organizing principle for online learning materials and we term coherently organized digital exercises and expositions. This organizing principle is larger in scale than individual lessons, or particular teaching interventions, but smaller than a programme of study (e.g., degree subject). It operates at a mesoscale and is increasingly guiding our thinking in the development of university mathematics courses. Furthermore, within our organizing principle, there is significant scope to implement recommendations which prior educational research has found to be effective. We are motivated to write this because we are becoming aware of a growing disjunction between (1) traditional ways of teaching and (2) the demands of our students and of the institutions in which we work. This disjunction has been thrown into sharp focus by the challenges of responding to COVID-19. We started our work, and writing this article, before March 2020. We hope this discussion will be helpful when deciding what to keep from the upheaval in education caused by the emergency response to the COVID-19 pandemic.
Coherently organized digital exercises and expositions (CODEX) are a sequence of interactive online quizzes which form the set of learning materials for a course.
Quite literally, we have taken the book and put it inside the quiz. That is, there are short blocks of text, worked examples, interactive diagrams, and short focused video clips embedded within the sequence of exercises and questions. Students work through the quizzes, and success in the course is judged by their success on the quizzes. To be clear: there are no textbooks, printed notes or .pdf documents; no regular live lecture events; no problem/exercise sheets to hand in (i.e., homework); and no separate final written examination. Thus, a CODEX provides a way to deliver a fully online course in mathematics [33].
The focus is firmly on exercises and questions which direct students' attention and require their activity. When students and instructors use these materials, there is certainly scope for the social contact which many people find vital for motivation and learning: coherently organized digital exercises and expositions are simply the materials provided by the instructor and the work expected by the instructor from the student.
In this paper, we articulate the rationale for developing such a principle, we specify the implementation details and discuss the issues involved. What we seek to do here is to name our principle, define it and place it firmly in view for discussion, debate and evaluation.
We are nervous in claiming novelty or priority for this idea. Indeed, the past 20 years have seen impressive major projects for online learning such as those reported in [1,34] and online initiatives such as Khan Academy. Some of these major initiatives have been successful, but many have required large capital development and have not sustained users beyond the initial development period. Our approach does require additional resources, which we discuss below. However, by working at the course level, we have developed a contemporary course format, rooted in theory, which is practical (demonstrably) by university instructors.

How People Learn
Why might we anticipate positive benefits from organizing a course around a CODEX? The design is derived from our understanding of how people learn.
Constructive alignment is one theoretical basis for designing teaching which is widely used in higher education [5]. The basic principle is to establish clear intended learning outcomes for a course and to address these directly through the teaching and assessment. So, for instance, if a learning outcome for a course is that students should be able to "calculate accurately" using certain techniques, then constructive alignment suggests that the teaching should involve the students in doing calculations (not just watching someone else do them), and the assessment should be designed to check that they can indeed "calculate accurately." Alongside this theoretical framework, there is a growing body of evidence that active learning approaches are preferable to traditional lecturing [14]. Findings from cognitive science perhaps offer an explanation for the effectiveness of active learning approaches. For instance, the testing effect is the well-established finding that "retrieval of information from memory produces better retention than restudying the same information for an equivalent amount of time" [28, p. 20]. This gives support to the idea of designing courses so that students are routinely retrieving from memory, as in active learning, and one way to do this is by mixing questions into the exposition [35]. The design of a CODEX achieves this by putting all of the textbook content into a quiz, so that exposition can be easily mixed with questions.

Increasing Student Numbers
Over the past 20 years, universities in the UK and elsewhere have seen a steady rise in numbers of students. Students arrive with a wider range of grades in mathematics, and the increasingly diverse international student intake have a range of qualifications from different school systems. In the UK, at least, there is also an increased focus on "student experience" with national initiatives such as the Teaching Excellence Framework and National Student Survey, both designed to focus institutional attention on developing and improving the quality of education. Part of our motivation for developing the CODEX is practical: we have to do the best we can to teach these students within the resources available.
In the 1980s, Bloom found that students taught by an individual tutor achieve test scores that are two standard deviations better than students who attend traditional classroom teaching [6]. A partial solution to the problem of finding a sufficient number of individual tutors is Learning for Mastery, where students are regularly tested by using formative tests and students are required to demonstrate a correct answer to 90% of the test problems, i.e., demonstrate "mastery." When a student falls short of mastery further teaching and testing is repeated, several times if necessary. Bloom's Learning for Mastery has been well-studied, with a number of independent studies reporting significant positive effects, e.g., [1], and see [17] for a review. Our approach to developing digital resources is strongly influenced by Bloom's Learning for Mastery. Bloom did not have the online assessment tools available to us, whereas coherently organized digital exercises and expositions implement mastery learning conditions in a practical way.

Contemporary Technology
During the past 10-15 years, educational institutions, particularly universities, have been gradually adopting more online technology to complement traditional oncampus teaching. This includes learning management systems, recording of live lectures, fully online courses and wider initiatives such as massive open online courses (MOOCs). A learning management system (LMS), sometimes termed a virtual learning environment (VLE), aims to provide a cohesive user environment with a single entry point accessed through a mainstream web browser. Most universities supply such a tool to support teaching.
In addition to communication and facilities for posting written lecture notes, slides and access to recordings of lectures, LMSs have an important role in orchestrating assessment activities. Students are increasingly being expected to use online assessment systems in traditional mathematics courses [29]. Publishers now routinely support traditional mathematics textbooks with such online assessment systems. The physical book still exists (and is often expensive to purchase), but the physical artifact is normally supported by a website which has interactive materials and online assessments. For the record, online assessment of mathematics goes well beyond multiple choice questions and for over a quarter of a century systems have accepted answers from students which are mathematical expressions [32]. A review of this topic is given in [29], and such systems are now beginning to support assessment of students' line by line reasoning, see [30] for more recent examples. It is our interest in automatic online assessment which has partly motivated this work, and we have a strong professional interest in this endeavor.
Note that we are not building another adaptive tutoring system. Adaptive systems, with complex user models, have proved very expensive to create, see [1]. We explicitly sought, and have developed, a design framework which is practical for a instructor to use in a wide range of situations.

Looking Forward
New technology normally looks back, seeking to automate current processes and replicating existing systems. For example, most professional mathematical writing is typeset using T E X (or L A T E X), software with a remarkably long period of continual use. The motivation of Knuth in developing T E X was to replicate traditional movable type printing in electronic form, as an aid to physical print publication [23]. Arguably, this was a missed opportunity to create a paradigm shift in mathematicians' behavior by harnessing the computer itself as an aid to encoding the meaning of an argument to help humans prove, check and search within their work. That change only came later, and is still gradual, ongoing, and by no means a universally accepted process [26]. A further example is provided by Babbage's mechanical computers. Babbage's motivation was to calculate tables of logarithms rather than providing individual users with the ability to calculate directly. 1 Again, there is no anticipation by Baggage of our ubiquitous ability to calculate. It is entirely natural that the first uses of educational technology would therefore almost exactly replicate traditional modes of teaching.
Much current educational technology is essentially designed to support traditional modes of teaching. That is, colleagues still give live lectures to large groups, and in mathematics lectures at universities this still involves exposition using chalk boards [3,16]. Assessment of university courses is similarly traditional and dominated by examinations: a recent UK survey found that over 25% of modules were 1 Certainly his motivation for the difference engines was to print logarithm tables, but even when describing his work on the Analytical Engine Babbage refers regularly to calculation of tables, and correction of errors in pre-existing tables. See, e.g., [4, p. 138].
assessed entirely by closed book examination and nearly 70% used closed book examinations for at least three quarters of the final mark [19]. Online assessment is widely used, but most often to replace traditional exercise sheets, which would previously be submitted to be assessed by hand. Our appreciation of the potential of a CODEX has been gradual. The idea of creating courses consisting entirely of exercises can be found clearly enacted in the books of Robert Burn, e.g., [7][8][9]. Books entirely based on exercises need not be a linear learning experience either, e.g., see [11]. So the idea of all activity being exercises for students to complete is certainly not new. Indeed, our ideas in this area formed recently following serious thought about how to teach computer programming to undergraduate students more effectively. The CodeRunner assessment system (http://coderunner.org.nz/ Oct 2021) was developed by Dr Richard Lobb and Dr Jenny Harlow. They have used CodeRunner to structure their computer programming courses around quizzes [24]. Our conversation with them both helped to crystallize our thinking on the subject and to increase our confidence to fully embrace the consequences of developing and running an undergraduate course based on a CODEX. Our first, tentative, steps in this direction came in 2017-2018 with the development of a new course for graduate students learning to programme in Python, but while this course made more extensive use of quizzes there were still many strong traditional elements. The design was further developed in 2018-2019 with the creation of the course described in the next section.

RESULTS: DEVELOPMENT AND PILOT OF FAC
Our university introduced the course Fundamentals of Algebra and Calculus (FAC) to address the variability of incoming students' prior knowledge. It is delivered in the first semester, with students advised to take it based on their grades from school and their performance in a short diagnostic test [20]. The course is notionally 200 hours of work for the student, counting for 20 credits out of 60 normally taken in a semester. There is only one 20 credit course (linear algebra) that is compulsory for mathematics students in the first semester of Year 1, so students electing to take FAC are taking it in place of other "outside" options.
The topics in the course are based on the content of typical high school syllabuses (in particular, SQA Advanced Higher Mathematics and GCE A-Level Further Mathematics), with a focus on calculus methods and supporting algebraic work. It is mainly aimed at increasing students' preparedness for the calculus course taught in the second semester. There is only a small overlap with the linear algebra course that students take in parallel with FAC, to give some extra practice with basic vector computations.
The topics are arranged into 10 weekly units, released each week during semester and with deadlines in the following week. Each weekly unit has a common structure: • a "Getting Started" section, which motivates the week's topic and reviews prerequisite content (e.g., differentiation facts when starting integration), • four sections of content, each of which is designed to take around 2 h to complete (roughly equivalent to one traditional lecture plus associated practice), • a 90-minute "Practice Quiz" with a mix of questions on the work of the week; this can be taken an unlimited number of times, with full feedback provided on each attempt, and • a 90-minute "Final Test" which is similar in style to the Practice Quiz, but only allows a single attempt.
All of these are implemented as quizzes in the Moodle virtual learning environment -in particular, the sections of content are a CODEX for the course. The Moodle quizzes make extensive use of the STACK question type, as well as the ability to embed other content such as videos and applets (e.g., GeoGebra and H5P). An example from one quiz is shown in Figure 1, and a full weekly unit is available at https://stack-demo.maths.ed.ac.uk/demo (Oct 2021).
A student's grade on the course is determined by combining the results of the 10 weekly Final Tests (together worth 80% of the grade) with a final 2-h synoptic test covering topics from the whole course (worth the remaining 20%). The course was well-received by students in its first delivery in the 2018/2019 academic year, and their overall performance on the course was strong: the mean student score was 66%, while 47% of students achieved an A grade and only 5/109 students failed. Since this course also includes a number of other initiatives motivated by educational research [21], we are undertaking a detailed evaluation of its effectiveness. Initial results suggest that FAC has been effective in closing an attainment gap, as measured by a pre/post diagnostic test [15]. Further work is ongoing, which will investigate the impact of FAC on performance in subsequent courses, and on attitudes toward mathematics study [22].

DISCUSSION
Our experience so far with coherently organized digital exercises and expositions has raised a number of issues which will warrant further thought and discussion.
First, a challenge shown by the first delivery of FAC is how to engage learners together with the CODEX -that is, how can the design also take account of the social aspect of learning? With FAC, students were left to work independently through the CODEX, with support available from a drop-in help centre and an online forum. Of course, some students may have arranged their own informal study groups to work together on the material, and one could imagine timetabling such sessions. Indeed, with FAC we set up "autonomous learning groups" which were groups of up to six students; each week we provided a rich task for the groups to work on, and invited each group to upload a record of their work.
A commonly raised concern with online assessment is the possibility that students will plagiarize, collude, or make use of tools such as WolframAlpha and thereby undermine the assessment [2]. We have taken the view that it is best not to focus attention on this aspect in our messaging to students. Instead, we emphasize how the skills developed in the course are connected and lead in to later courses. Certainly with FAC, the students will progress to a traditionally taught calculus course with a final written examination, so any student who does not apply themselves will ultimately come unstuck. Other courses based around a CODEX could of course choose to include invigilated assessments to address these concerns.
For students, it seems that working through the CODEX required a significant effort. Each weekly unit ended with an optional feedback form, with one question asking how many hours the student had spent working on the course that week. Across 173 responses received all semester, students reported a mean of 10.7 h of work per week. While this is, if anything, a little low relative to the notional hours of work expected, it should be noted that some students were reporting spending 20-30 h in a given week. In our subsequent interviews with students [15], concerns about perceived workload emerged as a theme. In our view, the workload is appropriate as "time on task" is important for learning -but we are mindful of the danger of an "assessment treadmill" which may supplant students' inherent interest in the subject with a desire for marks.
For staff, there is a substantial outlay of effort in preparing the material. We estimate that the development of FAC took approximately six working months. There is a potential concern here about how this work is valued/ recognized, particularly over the long term -a traditional lecture course needs a lecturer in post, but are we redundant when the CODEX is written [27]? Our view, based on experience so far, is no: the material will need annual expert review and student engagement should be monitored and supported by a subject specialist.
In common with many methods-based mathematics courses, FAC has a strong focus on developing procedural fluency in mathematics. This fits well with the existing capabilities of computer-aided assessment systems like STACK since they have well-developed tools for assessing basic algebra and calculus, though typically with a reliance on the final answer rather than the solution steps [13]. The extent to which these tools can be used to teach and assess more abstract conceptual knowledge, including reasoning and proof, is less clear. One approach is alternative assessment design such as proof comprehension activities, see [31]. In parallel new features are also being developed, e.g., assessing line-by-line reasoning, which take the first steps toward automatic assessment of a complete proof [30].
The current CODEX design is linear: all students work through the same set of material in the same order. We regard this as a strength, as it greatly simplifies the authoring task in comparison to producing adaptive learning materials. Previous studies, such as [25], have been cautiously optimistic about the effectiveness of computer based instruction in general. The Achilles' Heel of the intelligent tutoring system (ITS) approach is the need to externally construct the domain representation in advance, for instance with Bayesian Knowledge Tracing which relies on a view of skill acquisition that atomises knowledge into isolated skills [10]. Our goal is not to replace a human instructor with an ITS, but re-position the instructor's role in the educational process. Indeed, after a decade of work, [1] acknowledged that they had "totally abandoned our original conception of [automatic] tutoring as human emulation" and conceded that developing an ITS is an expensive and expansive undertaking. More importantly, we believe the philosophy behind ITS ossifies knowledge and disempowers instructors. However, it does raise the question of how to support students when they become stuck on the linear path; in FAC, we provided students with access to face-to-face help in a drop-in help centre, and with an online discussion board.
We set our students a challenging week-by-week timetable to complete work (within a context of university term dates), but there is also potential for self-paced use as well as distance learning.
While our experience is derived from teaching adults at university, there are similar and parallel developments to provide materials to students at school. Homework systems and traditional publishers are increasingly relying on online materials for school students as well.

CONCLUSION
We are living through a revolution, and education will be changed as a result. Our design principle: coherently organized digital exercises and expositions, putting the "book inside the quiz," provides a clear way to think about the nature of online learning. This is also an opportunity to embrace the potential that new technology has to implement evidence-based practices such as spaced retrieval practice [18].
Prior to COVID-19, teaching in universities had yet to see the sort of disruption which is taking place in the music industry, media and publishing. In 2015, Deming and colleagues suggested that "The rapid growth of online for-profit education during the past decade and the recent entry of flagship public universities into the sector suggest that the competition for online students in higher education has only just begun" [12]. It transpired that it was not so much competition but necessity which has accelerated the adoption of online resources. The period March 2019, up to the time of writing (Oct 2021), has seen a period of unprecedented rapid change in the way teaching has taken place. Lock-downs, or other legislative stay at home orders, forced almost all teaching online and hence all teaching colleagues to engage with tools and techniques for using those tools. Much of what has happened since March 2019 has been an emergency response, and it remains to see what we will decide to keep and what to reinstate.
Coherently organized digital exercises and expositions seek to look forward, using established educational research results and harnessing the full facilities of contemporary technology. We are changing the model of education for university mathematics courses in substantial and non-trivial ways. Some of this design (e.g., constructive alignment) is independent of the format, whereas the focus on mastery using automatically assessed quizzes, adjacent worked examples and retrieval practice is only possible with the format we have chosen (or something similar). The conscious combination of features represents a substantial piece of educational design. We accept there is potentially a very subtle difference between putting a "quiz at the end of a section," and embedding "material at the start of a quiz," although the latter is actually a pastiche of what we have developed. Our point here is in what, exactly, is the top level organizing unit around which the students' activity is organized. We now have a clarity of purpose when choosing quizzes (formative and summative) as the primary organizing unit within a course. The doctrine of marginal gains suggests that progress is made through the accumulation of conscious decisions, which together create a significant improvement. We believe coherently organized digital exercises and expositions is one such incremental gain, and worthy of a name and of discussion. Further research, discussion and experimentation, will help us understand whether these initiatives are ultimately effective.
As a response to COVID-19, and while this paper was under initial review, we developed a further CODEX with colleagues in Edinburgh which is the natural prequel to FAC, within the UK system. During 2021-2022, Introduction Mathematics and its Applications (IMA) is having its second student presentation, and we are now developing a further CODEX teaching statistics to non-specialist students (i.e., not mathematics majors). We are also informally working with colleagues at other UK institutions to discuss the design of courses along similar lines elsewhere. We would welcome correspondence with colleagues interested in developing similar courses, whether in mathematics or other disciplines. The COVID-19 pandemic is likely to create some lasting changes in education, and with hindsight may prove to simply accelerate the changes to which we were already responding.

DISCLOSURE STATEMENT
No potential conflict of interest was reported by the author(s). His learning and teaching interests include all aspects of automatic assessment of mathematics using computer algebra and online learning. Chris Sangwin is the author of a number of popular science books and he has also co-authored a number of school textbooks published by Haese Mathematics.

ORCID
George Kinnear is a lecturer in technology enhanced mathematics education in the School of Mathematics at the University of Edinburgh. He completed both his undergraduate degree in mathematics and Ph.D. in pure mathematics at the University of Edinburgh. His research interests are in effective uses of technology to support undergraduate mathematics teaching, with a particular focus on the use of computer-aided assessment.