Using a lecture-oriented flipped classroom in a proof-oriented advanced mathematics course

ABSTRACT In traditional advanced mathematics lectures the instructor usually provides definitions, theorems, and proofs on the board rather quickly. The students often cannot make sense of these during the lecture as they are busy writing. In order to gain an understanding of the content, an intensive post-class processing on the basis of their notes would therefore be necessary for many students. However, students often do not post-class process lectures carefully. Furthermore, they often do not record the lecturer’s oral explanations that are essential to make sense of the formal lecture content. One approach to address these problems is a flipped classroom, in which students come into contact with new content before class, while the in-class time is used for activities that help students make sense of it. We implemented an adaptation of this approach in a proof-oriented analysis course, in which the instructor used the in-class time to illustrate how experts in mathematics make sense of new definitions, theorems, and proofs, while keeping the lecture as the mode of teaching. In this article we describe this adaptation, its possible benefits for addressing students’ problems in learning from traditional mathematics lectures, and its actual effects, which we investigated in an empirical study.


Introduction
Lectures at university are often criticized for turning students into passive listeners (Bergsten, 2007;Yoon et al., 2011). However, lectures also have some advantages (Pritchard, 2010). First, the lecturer, as an expert in the discipline, can identify and communicate the structure and core ideas of the subject. Furthermore, the lecturer can model the behaviour of experts in the subject. In particular, she/he can illustrate how such experts make sense of the new content covered during the lecture. But the lecture format can also help to motivate students for the subject because the instructor's enthusiasm for it can be passed on to students. Rood (2003) even compares lectures to a theatre in which the lecturer can induce excitement and awe towards the subject with his/her performance. In addition, lectures can cultivate a group dynamic: the students might feel that they are a group of learners sharing the same experience, which can also raise students' motivation for the subject. Finally, the lecture format requires relatively few resources. In particular, a lecture can be delivered simultaneously to many students.
However, learning from lectures is not trivial for students as this requires a range of skills (Lew et al., 2016;Williams & Eggert, 2002), namely (1) listening and paying attention, (2) cognitive processing, (3) note-taking, and (4) reviewing. Problems concerning each of these aspects are documented in the literature. First, students in mathematics lectures often cannot be attentive towards the lecturer's explanations as they are too busy copying the definitions, theorems, and proofs from the board (Freitag, 2020). This can make it hard for them to process information presented by the instructor and to gain an understanding of the content during the lecture because the instructor's explanations often contain information that is essential for gaining an understanding of the content, like informal representations of a concept or ideas in a proof (Fukawa-Connelly et al., 2017;Lew et al., 2016). Therefore, an intensive post-class processing of mathematics lectures is necessary for many students to make sense of the content. However, students often do not post-class process lectures carefully (Bauer, 2015;Göller, 2019). Furthermore, they often do not record the lecturer's informal oral explanations (Fukawa-Connelly et al., 2017), so that even if they invest a considerable amount of time for post-class processing, it may still be hard for them to make sense of the formal content (the definitions, theorems, and proofs) by themselves after class solely on the basis of their notes.
One approach to address these problems might be the use of a flipped classroom, in which students come into contact with new content before class while the in-class time is used for activities that might help students make sense of it. In proof-oriented advanced mathematics courses, however, the content is often so difficult that a strong input from the lecturer, as an expert in mathematics who is able to communicate core ideas behind the definitions, theorems, and proofs covered as well as the dialectic between the formal and the genetic structure of the subject (Pritchard, 2010), might be beneficial for students to make sense of the formal lecture content. We therefore implemented a more lectureoriented adaptation of the flipped classroom approach into an analysis course, in which the instructor used the in-class time to illustrate how experts in mathematics make sense of the new definitions, theorems, and proofs that the students have acquainted themselves with before class.
In this article, we first describe this lecture-oriented adaptation of the flipped classroom in detail and illustrate its potentialities for addressing students' problems in learning from traditional advanced mathematics lectures. In the second part, we present an empirical study that investigates to what extent these potentialities could be actually realized in the proof-oriented analysis course where we implemented this flipped classroom approach. The findings are particularly relevant for teachers who wish to support students' learning while maintaining the lecture as the mode of teaching. Furthermore, the findings might be interesting for future research, as they suggest that further modifications are needed so that the possible benefits of the lecture-oriented flipped classroom might be fully realized.

Students' difficulties in learning from mathematics lectures
As already stated, students need to perform several activities to successfully learn from lectures (Lew et al., 2016), namely (1) paying attention, (2) cognitive processing of the information presented, (3) note-taking, and (4) reviewing. Concerning each of the four activities, several difficulties are documented in the literature.
First, it was already being shown in the '70s for lectures in various subjects that students often cannot maintain attention during lectures (Bligh, 2000). Although we did not find an empirical study measuring students' attention in mathematics lectures explicitly, a decrease in attention can also be expected there due to a high speed of the lecturer (Harris & Pampaka, 2016), and to a low level of interaction (Yoon et al., 2011). Of course, this does not mean that in general no interaction takes place in lectures, but the findings in the literature suggest that in traditional mathematics lectures it might be hard for students to maintain their attention.
The high speed of mathematics lectures might also make it difficult for students to process information presented (Harris & Pampaka, 2016). In traditional advanced mathematics lectures, whose content mainly consists of formal definitions, theorems, and formal proofs, making sense of the information presented during the lecture might be even more difficult. If a mathematical concept is merely introduced by its formal definition, students may have difficulties in making immediate sense of it because an understanding of a concept does usually not emerge from its formal definition (Tall & Vinner, 1981). And if a proof of a theorem is simply presented as a chain of statements that are deduced from one another, it may be difficult for students to develop an understanding of the proof as a whole, because the comprehension of proofs in advanced mathematics also has a global dimension that requires students to grasp the overarching idea of the proof (Mejia-Ramos et al., 2012).
Third, it is documented in the literature that taking adequate notes is not a trivial issue for students. Locke (1977), for example, ascertained for lectures in various subjects that students often miss information delivered by the lecturer. Fukawa-Connelly et al. (2017) found, specifically for advanced mathematics lectures, that students often do not record the lecturer's spoken comments. This is an important problem in such lectures, in which mostly only the formal content (definitions, theorems, and proofs) is written on the board, but additional information that is essential to make sense of this content (like informal representations) is often just provided orally. Van Meter et al. (1994) found, in a large study with 252 participants studying various subjects, two main reasons why students do not record important information: the high speed of the lecturer's delivery and a vague structure of the lecture. In particular the first issue is problematic in mathematics lectures, which often proceed very rapidly (Harris & Pampaka, 2016). Furthermore, Iannone and Miller (2019) found specifically for mathematics lectures that students do not record oral comments of the instructor because they do not consider these as relevant for the exam.
Finally, also concerning students' review of mathematics lectures, problems are documented in the literature. Students often focus their work out of class on homework problems they are required to solve (Göller, 2019). In advanced mathematics courses these problems often have the purpose of elaborating students' understanding of the mathematical theory covered in the lecture, and already require an initial grasp of the concepts, theorems, and proofs presented there (Bauer, 2015). Students are therefore usually expected by instructors of such courses to post-class process the lecture before being able to solve their homework problems (Bauer, 2015). According to our experience, however, it mostly remains implicit which activities students could (and are expected to) perform during this post-class processing phase in order to make sense of the definitions, theorems, and proofs of the lecture. Hence, students might not know how they could process a lecture after class to gain comprehension of its content.

Approaches for overcoming these difficulties
There are several approaches documented in the literature that might help address these problems. Due to limited space, we briefly mention of three of these.
One approach that can address the problems mentioned in the last section is the use of video presentations -either instead of or as a complement to live lectures. Yoon et al. (2014), Dunn et al. (2015), and Kinnari-Korpela (2015), for example, supported this hypothesis empirically by surveying students in terms of how they learned mathematics/statistics in a course using video presentations. What students in their studies particularly appreciated about video presentations was their freedom to pause a presentation if they felt they needed to think more deeply about the lecturer's explanations. Hence, the use of video presentations can help address the problem that students often cannot process information presented in live lectures because they cannot follow the instructor. Students also appreciated that they were free to view these video presentations again at a later point. This could address the problem that students often do not record the lecturer's oral comments in advanced mathematics lectures because they have access to these all the time. Finally, being able to pause might aid students in taking suitable notes because they are less likely to miss important points due to the instructor's speed.
Another approach for addressing the problems mentioned in Section 2.1 is to provide students with full or partial lecture notes before class (Cardetti et al., 2010;d'Inverno, 1995;Iannone & Miller, 2019). In particular, the effects of the use of guided notes, which are preprinted lecture notes with blanks at certain places, on students' learning in mathematics lectures are of interest in recent research in mathematics education. Iannone and Miller (2019), for example, found in a study investigating students' attitudes towards guided notes that these types of notes allowed them to also be attentive to the lecturer's oral comments. Furthermore, students in the study by Iannone and Miller (2019) claimed that the use of guided notes also allowed them to note down some of the instructor's oral comments. Since these often contain information crucial for making sense of the definitions, theorems, and proofs covered, the possibility of including these in one's own notes might be particularly beneficial for a student's review phase after a lecture or for revision before the exam.
The approach we chose to address the problems mentioned in Section 2.1 was a flipped classroom. The idea behind this is to switch the activities that are traditionally done in class now out of class, and vice versa (Lage et al., 2000). This means particularly that the introduction of new content takes place at home before class. This can, for example, take place with the aid of video lectures, (e)books, and online quizzes (Cronhjort et al., 2017;Jungić et al., 2015;Lai & Hwang, 2016;Maciejewski, 2015;Murphy et al., 2016). The inclass time can then be used for activities that can promote an understanding of the content. Examples of such activities documented in the literature (Cronhjort et al., 2017;Jungić et al., 2015;Maciejewski, 2015;Murphy et al., 2016;Petrillo, 2016;Wasserman et al., 2017) are: (1) the use of clicker questions with subsequent peer-discussions, (2) (group) work on quizzes and tasks, (3) examining possible applications, (4) students' presentations, (5) but also teacher-driven discussions of difficult material.
The literature suggests that the flipped classroom can especially address students' difficulties to stay alert in class (Jungić et al., 2015). Furthermore, Wasserman et al. (2017) showed that students communicated more in a flipped calculus III course than in a comparable traditional lecture. Such a higher level of in-class communication can in particular help students make sense of the content of the lecture (Jungić et al., 2015).
Since the content in proof-oriented courses is often very difficult, and students probably need a strong input from the instructor to master it, we consider a lecture-oriented adaptation of the approach as promising for such courses, which we implemented into a proof-oriented analysis course. We will now describe this adaptation and its potentialities for addressing the problems mentioned in Section 2.1.

Description of the lecture-oriented adaptation of the flipped classroom we used
According to the idea of the flipped classroom, the students first come into contact with the definitions, theorems, and proofs covered before class (for example on the basis of a script). The instructor then uses the in-class time to provide additional explanations of these, and to implement activities illustrating explicitly how to make sense of these as an expert in mathematics. With these activities the instructor could help students to also make sense of the definitions, theorems, and proofs covered. Furthermore, the instructor shares his/her expertise in how to gain an understanding of new definitions, theorems, and proofs in general with the students. This might especially show students how they could post-class process (also other) traditional proof-oriented mathematics lectures.
Concerning the concepts introduced by formal definitions, the instructor can, for example, illustrate during the lecture how mathematics experts make sense of such definitions, for instance by considering examples, counterexamples, or visualizations of the respective concepts, or by connecting these with other concepts introduced previously (Shepherd & Van De Sande, 2014). These activities can in particular help students develop a valid concept image representing the total cognitive structure associated with a concept (Tall & Vinner, 1981). Similarly, the lecturer can illustrate how mathematics experts make sense of a theorem, for instance by considering examples that fulfil the theorem, by considering counterexamples, by visualizing the statement of the theorem, or by using the theorem to solve certain problems. This might again help students develop a rich mental structure concerning the theorems introduced that is called statement image in the literature (Selden & Selden, 1995).
With regard to the proofs covered, the instructor can initiate activities that illustrate how experts in mathematics gain both a local and global understanding of such formal proofs (Mejia-Ramos et al., 2012). This can, for example, cover the following activities: (1) making sense of the symbols and individual statements in a proof, (2) discussing explicitly arguments for each statement in a proof (often called warrants) (3) filling gaps in a proof, (4) discussing the idea of a proof, or (5) constructing analogue proofs.
The in-class activities mentioned above do not of course need to be solely demonstrations by the lecturer. The students could also be included in these, if possible, for instance by encouraging them to find examples of concepts fulfilling a formal definition introduced or to identify warrants in the steps of a proof by themselves or in a discussion. Nevertheless, the adaptation of the flipped classroom just described (which we consider as promising for proof-oriented advanced mathematics courses) is more teacher-centered than other implementations we have so far found in the literature (see Section 2.2).

Potentialities of the flipped classroom in this adaptation to address students' difficulties
First, the problem of decreasing attention and students' difficulties in processing information presented during the lecture could be addressed. Since students become acquainted with the new definitions, theorems, and proofs before class, these no longer need to be put on the board and copied by the students during the class itself. Students could hence become more attentive towards the instructor's explanations, which they often cannot grasp in traditional lectures as they are busy writing (Freitag, 2020). On the other hand, the lecture-oriented adaptation of the flipped classroom just described still makes use of the instructor's expertise to communicate the structure and core ideas of the subject during her/his illustrations of how mathematicians make sense of new definitions, theorems, and proofs. This might be particularly helpful for students to gain an understanding of this formal content because the instructor can well illustrate the dialectic between the formal and the genetic structure of mathematics (Pritchard, 2010).
Second, since the in-class time is used for additional explanations that aim at promoting an understanding of the formal lecture content (definitions, theorems, and proofs), it is to be hoped that students also note down some of these, so that their notes do not contain only the formal definitions, theorems, and proofs as in a traditional lecture (Fukawa-Connelly et al., 2017). This is important if students wish to make sense of the content later again, for example when preparing for the exam.
Finally, the approach could also address problems concerning students' post-class processing of mathematics lectures. If students already gain an understanding of the formal lecture content in class, an intensive post-class processing is no longer necessary, which can address the problem that students often do not post-class process mathematics lectures intensively (Bauer, 2015;Göller, 2019). Furthermore, the instructor's in-class activities illustrating how experts in mathematics make sense of the definitions, theorems, and proofs covered could provide students with meta-knowledge of how they could post-class process a traditional advanced mathematics lecture, which remains often implicit. This is a particular benefit of the lecture-oriented adaptation of the flipped classroom we used, which might reach beyond the confines of the particular course in that it is implemented.
Nevertheless, the flipped classroom also has a disadvantage. The students need to invest time -and maybe additional time -to acquaint themselves with the new definitions, theorems, and proofs before class. If they do not, then the in-class activities might be meaningless for them.

Overall structure of the course
We implemented the approach into a proof-oriented analysis course, in which the second author was the lecturer herself. The participants were prospective mathematics teachers for grammar schools. Grammar schools cover class grades 5-12 or 7-12 (depending on the region) and have an academic profile that aims at preparing students for university. About 130 participants were enrolled in this analysis course, which consisted of two 90-minute lectures and one 90-minute tutorial per week. Furthermore, the students were required to spend time out of class in reviewing and working once more through lecture content (seven hours each week according to the curriculum), and for solving homework problems that were graded (six hours per week). For admission to the final exam the students needed to attain 60% of the possible points in the homework problems.
The following content was covered in the course: foundations of logic and set theory; rational numbers and their properties; sequences of rational numbers (including the formal limit concept); the construction of the real numbers as equivalence classes of rational Cauchy sequences; complex numbers; sequences and series of real and complex numbers; continuity; construction of the exponential and trigonometric functions with series; and one-variable differential calculus.
One session per week took place as a traditional lecture, while the other was held as a flipped lecture as described in Section 3.1. Students therefore experienced both lecture types in the same course with the same instructor. We will now describe the implementation of the two lecture types in detail.

Before-class activities of the flipped lectures
For the flipped lectures, the students received a script containing all the definitions, theorems, and proofs covered in advance. A sample page on the topics 'uniform continuity' and 'Lipschitz continuity' is shown in Figure 1.
As can be seen there, the script in fact only contained the formal content (definitions, theorems, proofs) and some examples. The students were then asked to (1) read through the script and copy it, (2) try at least to make rough sense of the definitions, theorems, and proofs covered if possible, (3) memorize the definitions before class. Letting the students perform these activities had the aim that the students would become as familiar with the definitions, theorems, and proofs as they would during a traditional mathematics lecture, in which students copy these from the board, and try to make initial sense of them if possible. This way, the students would hopefully already bring some familiarity with the content for the actual deep engagement with it in class. The instructor especially hoped that if the formal definitions had already been memorized before class, then students would be able to connect these more easily to the concept image, whose development was fostered in class. To motivate the students to genuinely perform the activities above, an online quiz was provided in which students were given questions which they should already be able to answer after they read the script, even if they have not fully understood its content. Two sample questions are shown in Figure 2. One of these asks for properties given in the definition of an equivalence relation, while the other asks for notations. Since both had been explicitly given in the script, students could choose correct answers if they had just read the script (and memorized the definitions), but maybe not yet had acquired a full understanding of its content.

In-class activities of the flipped lectures
During the flipped lectures the instructor illustrated, on the basis of her own expertise as a mathematician, how she would make sense of the definitions, theorems, and proofs in the script. To structure the lecture, the instructor put those aspects of the script, which she wanted to discuss further during the lecture, on slides (mostly the definitions, theorems, and proofs; for an example see Figure 3). For each of these aspects, she implemented several activities in class that she would carry out to acquire an understanding of these, which took place on the board. If possible, she implemented the activities as a dialogue between herself and the students to better maintain students' attention.
Most of the activities in the flipped lectures focused on promoting an understanding of the concepts introduced by formal definitions and of the proofs laid out in the script. As already mentioned, regarding the concepts introduced by formal definitions, the instructor discussed examples, counterexamples, and visualizations in class. Concerning the definition of uniform continuity on the sample page in Figure 1, for instance, she discussed the function f (x) = √ x as an example and f (x) = 1 x as a counterexample to promote the development of a valid concept image of the concept (see Section 3.1). Furthermore, she provided visualizations illustrating why the functions either fulfil the definition of uniform continuity or do not (see Figure 4): for f (x) = √ x the same δ can be used for all x to stay in the -neighborhood, while for f (x) = 1 x the δ required becomes arbitrarily small for x → 0. With regard to Lipschitz continuity, she did the same, but for the function f (x) = |x| as an example and for f (x) = √ x as a counterexample. In both cases, she emphasized that students should always try to find examples, counterexamples, and/or visualizations for definitions presented in mathematics lectures during their own post-class processing. She then explained that these examples already suggest a hierarchy of the different notions of continuity so as to connect these new concepts to the overall notion of continuity that had been introduced in the last session, and she recommended that students should visualize such connections by means of a concept map (see Novak and Cañas (2006) for further details about concept maps). Similar activities also took place for other mathematical concepts introduced by formal definitions and for the theorems covered.
For helping the students gain an understanding of the script proofs, the instructor implemented several activities that illustrated what she herself does to get a local and global understanding of such script proofs, and that aimed at promoting the development thereof among the students. Regarding a local understanding of these proofs, she especially discussed (1) the meaning of the symbols and the individual statements in the proofs, (2) the warrants in the steps of the proofs, i.e. the arguments for the justification of the corresponding statements (3) gaps in the proofs of the script.
These activities mostly took place in the form of a dialogue between instructor and students. In the case of the proof of the hierarchy of the different notions of continuity from the sample page (Figure 1), for instance, the instructor discussed with students why continuity can be deduced from uniform continuity on a formal level by comparing the two definitions and discussing the meaning of the order of the quantors. Concerning the claim that Lipschitz continuity in turn implies uniform continuity, she highlighted that the script proof in Figure 1 has the gap to show that δ := ε L really meets the requirements for uniform continuity. She then developed the corresponding argument step by step on the board on the basis of students' inputs (including a discussion of the warrants for each step).
Besides these three activities described above, the instructor also illustrated how to get a holistic understanding of the script proofs in the flipped lectures, for example by (1) showing the development of the script proofs and strategies for how to reconstruct these (example: illustration of how to find a δ for a given ε in limit proofs), (2) explaining the ideas behind the proofs (example: using powers to construct a geometric series as a majorant for the proof of the root test for the convergence of series), (3) constructing analogue proofs on the board in dialogue with the students (example: proving claims for monotone decreasing functions in class if they had been proven for monotone increasing functions in the script).
Overall, the instructor designed all these activities to illustrate how she herself would try to make sense of the definitions, theorems, and proofs given in the script. Hence, she heavily relied on her expertise as a mathematician. Nevertheless, the activities also match with important findings concerning the development of an understanding of mathematical concepts, theorems, and proofs documented in the literature in mathematics education (see Section 3.1).
During the activities of the flipped lectures the instructor always emphasized that such activities are important for being able to gain an understanding of formal definitions, theorems, and proofs, and emphasized that students should also perform such activities for themselves during their post-class processing of a traditional lecture.

The traditional lectures
The traditional lectures were held as chalk talk, i.e. the instructor wrote the definitions, theorems, and proofs on the board, and gave additional oral explanations (Artemeva & Fox, 2011). To help students tackle the formal definitions, she also provided examples, counterexamples, and visualizations. However, there were two differences compared to the flipped sessions: (1) The instructor provided all of these by herself and did not involve the students in a discussion.
(2) She did not make explicit the necessity to consider such for gaining an understanding of formal definitions during the post-class processing if these had not been provided within the lecture.
With respect to proofs, the instructor presented these in the traditional lectures in the form of a chain of arguments on the board. She mentioned warrants by herself and did not involve the students in a dialogue or discussion. Furthermore, she did not explicitly discuss possible gaps within the proofs she presented, so that the students were required to remedy these by themselves during the post-class processing. A contextual difference concerning the treatment of proofs in the traditional sessions was that the instructor often could not include activities designed to foster a holistic understanding of the proofs, for example the construction of analogue proofs, due to time limitations.
Nevertheless, despite the contextual difference just mentioned, the main difference between the two lecture types was that in the traditional sessions the instructor tried to help students gain an understanding of the formal lecture content by just presenting additional explanations, such as examples for definitions or warrants in a proof, while in the flipped sessions she made explicit for the definitions, theorems, and proofs the students had become acquainted with before class how the students could make sense of these by themselves during the post-class processing of an advanced mathematics lecture in which only formal content is presented.

Methodology of an evaluation of the lecture-oriented flipped classroom
We investigated empirically to what extent the lecture-oriented flipped classroom we implemented might be helpful for students' learning compared to a traditional lecture, and to what extent the benefits hypothesized in Section 3.2 to address students' difficulties in learning from traditional mathematics lectures actually occurred. To answer this, the first author designed a questionnaire and gave it to students participating in the analysis course described in Section 3.3. As previously explained, this course consisted of two lecture sessions per week, of which one session was held as a flipped lecture while the other session was held as a traditional 'chalk talk' lecture. Hence, the students had experienced both lecture types in the same course with the same instructor.

Design of the questionnaire
The design of the questionnaire was based on a framework by Williams and Eggert (2002). It has its origin in research on students' note-taking in lectures, and was first used by Lew et al. (2016) to investigate students' learning from mathematics lectures. According to these authors successful learning from lectures requires four activities: (1) Paying attention (2) Cognitive processing of the information presented (3) Note-taking (4) Reviewing/Post-class processing The literature review in Section 2 showed that students have difficulties in performing these activities in traditional mathematics lectures. In the questionnaire, we asked to what extent the students (could) perform these activities in the two lecture types they had experienced in their analysis course in order to compare between flipped and traditional lectures. The questionnaire therefore consisted of the following four blocks of Likert items (the complete list of items is provided in Appendix in Figure A1): (1) Items focusing on students' attention during the lecture (Items 1-4) (2) Items focusing on students' information processing during the lecture (Items 5-10) (3) Items focusing on students' note-taking (Items 11-17) (4) Items focusing on students' post-class processing of the lecture (Items 18-25).
The answer format was always a Likert scale ranging from 1 ('Totally disagree') to 6 ('Totally agree'). The students were asked to answer each of these items for the flipped lecture and the traditional lecture. Since the flipped sessions also required a preparation, we additionally designed items that focused only on the preparation of the flipped sessions (see Figure 5).
To ascertain further the extent to which students occupied themselves with lecture content out of class, we also asked for the time they spent per week for their post-class processing of the traditional lecture, their preparation for the flipped lecture, and their post-class processing of the flipped lecture.
In addition, to find out if we had missed possible benefits of the flipped or of the traditional lecture for students' learning, we also asked the students which lecture type they preferred (answer options: 'flipped lecture', 'traditional lecture', and 'does not matter'), followed by an open question asking them to explain why.
Finally, we investigated if the in-class activities of the flipped lectures influenced students' post-class processing of a traditional lecture (the final benefit hypothesized in Section 3.2) with the following question: Have the activities during the flipped lecture influenced the way you post-class process a traditional mathematics lecture? If so, how?

Data collection
The questionnaire was administered to students during a lecture, about four weeks before the end of the semester. 98 students took part in the survey (of around 110 course participants at the time of the survey). Students who had previously dropped out were not reached due to data protection considerations.

Data analysis
Analysis of the Likert items: For checking the reliability of our measurements we posed different items for each of the activities for successful learning from lectures according to Lew et al. (2016), which we grouped together in scales. We then determined the consistency measure Cronbach's α (after reversing negatively-worded items), which yields a lower bound for the reliability of the scales (Krauth, 1995). The values of α are shown in Table 1. Although there is no consensus regarding a lower bound for α to consider a scale as reliable, 0.7 or 0.6 are often mentioned (Taber, 2018). Our values of α are within this range, indicating that the reliability of our scales is acceptable. We excluded one item (item 14 'I highlight issues not understood in my notes') of the block 'note-taking' from the further analysis because students' answers were neither consistent with the ones from the scale 'Noting additional comments' nor with the ones from the scale 'Taking suitable notes during the lecture'.

Analysis of the open questions:
We categorized the responses to the open questions that asked for a reason for the preference of one of the two lecture types, and to the question of how the activities of the flipped lectures influenced students' post-class processing of a traditional lecture, as follows: (1) Development of a first version of the category system on the basis of the data by the first author (2) Categorization of the data into these categories by the second author (3) Discussion about different codes between the two authors and agreement For all open questions, an answer could be assigned to multiple categories (for instance, several different benefits of the flipped lectures). We should also mention that, although we categorized the answers on the basis of the data, we already had in mind the framework we used for the questionnaire construction (the activities necessary for successful learning from lectures) while categorizing.

Results of the Likert items focusing on the activities for successful learning from lectures
As described in Section 4.1, the questionnaire first contained the Likert items focusing on the four activities of our questionnaire framework that are important for successful learning from lectures, namely (1) paying attention, (2) cognitive processing of the information presented, (3) note-taking, and (4) post-class processing. These Likert items resulted in five scales (Table 1). The corresponding distributions are shown in Figure 6. Non-parametric Wilcoxon tests showed that the distributions of the scales 'Information processing' and 'Noting additional comments' differed significantly between the two lecture types at an α level < 0.01 (p < .0001 and p = .0005), while the distributions of the other scales did not. Figure 6 and these tests show several results. The first result is the significant difference between the distributions of the scale 'Information processing'. This result is not surprising because in traditional mathematics lectures students have to concentrate very much on copying definitions, theorems, and proofs from the board, and are less likely to be able to process information presented.
The second result is that students were more likely to note down additional comments in the flipped sessions (and the corresponding distributions differed significantly, p = .0005), for example the instructor's comments (see the items in Figure A1). However, the difference between the distributions of the two lecture types in the scale 'Noting additional comments' is not substantial. Hence, it is not enough just to give students time, when they do not need to concentrate on copying content from the board, in order to encourage them also to note additional comments besides what the instructor writes on the board.
A third interesting result is that students' attention was not significantly different (p = .33) in the two lecture types although the literature suggests that students' attention is higher in a flipped classroom. However, this result can be explained by the fact that our adaptation of the flipped classroom was more lecture-oriented (see Section 3.1). Table 2. Results of a sample item concerning students' post-class processing of the two lecture types, with answers ranging from 1 = 'Totally disagree' to 6 = 'Totally agree' (N = 98). I fill gaps in the proofs that had been left in the lecture at home for myself. The final result from Figure 6 concerns students' post-class processing. We expected that they would have post-class processed the traditional sessions more intensively because fewer activities that promoted an understanding of the formal content were implemented in these. However, many students did not intensively post-class process the lecture for any of the two types, and the distributions of the corresponding scales did not differ significantly (p = .52). Table 2 illustrates this phenomenon for the activity of filling gaps in proofs, which is of particular importance for gaining an understanding of the proofs covered. Concerning the traditional sessions, this is a problem, because the instructor could not explicitly focus on gaps in the proofs presented there (see Section 3.3). Hence, incorporating activities into class that help students understand the formal content during the lecture can address the problem that students often do not perform such during their post-class processing.
In the case of the flipped lectures, students' engagement with the content out of class also took place before class. The mean of the 6-point Likert scale constructed from the corresponding items in Figure 5 was 3.0. This indicates that on average the students did prepare the flipped sessions, but not very intensively. However, we do not consider this a problem because the activities for gaining an actual understanding of the content took place in class in the case of the flipped lectures.

Results concerning the time spent out of class
Our data indicate that students spent more time with the lecture content out of class apart from solving the homework problems in the case of the flipped lectures. First, the time that students spent out of class per week examining the content of the weekly traditional session, determined as the time spent on post-class processing of the session, is displayed in Figure 7.
The instructor told the students that they were required to spend about two hours for the post-class processing of a 90-minute lecture, which make up part of the seven hours they have to spend with the lecture content out of class per week according to the curriculum (see Section 3.3). Hence, in the case of the traditional session, students spent on average only half of the time required.
The time spent per week for the preparation of the flipped session and the total time spent per week with the content of this session (time for preparation and post-class processing) are shown in Figure 8. This indicates that students engaged more with the content out of class with respect to time for the flipped sessions (about twice as much: compare Figure 7 with the right-hand graphic in Figure 8). In particular, students spent on average the required 120 minutes in the case of the flipped sessions. Of course, since these times are self-reported, the results are probably slightly biased.

Results of the open questions concerning perceived advantages of both lecture types
To find out if we had missed important benefits of the flipped or the traditional lectures, we asked the students which lecture type they preferred and why. The preferred types are shown in Table 3.
This indicates that the majority preferred the flipped lecture. However, there was also a substantial proportion who preferred the traditional lecture. Hence, neither of the lecture types suited everyone. To identify possible reasons for students' preferences, we then asked for these in a subsequent open question (see Section 4.3), and categorized the responses. We will now present the categories and interpret these from the perspective of our framework   for the questionnaire construction (the four activities that are important for successful learning from lectures according to Lew et al. (2016)).

Reasons for preferring the flipped lecture:
Of the 50 students who preferred flipped lectures, 49 gave a reason. As described in Section 5.3, we categorized the reasons mainly on the basis of the data, but with the activities of our framework for the questionnaire construction in mind. The analysis led to six categories, each of which included at least 4% of the course participants. An answer could also be assigned to multiple categories if students mentioned several benefits simultaneously. The categories and the proportion of the participants who referred to these categories are shown in Table 4.
The advantage of the flipped lectures most often mentioned was that the initial contact with new content took place at home before class (category 1 in Table 4), which is the main idea underpinning the flipped classroom. This had several benefits that became visible in the individual answers, and that can be seen in the examples in Table 4. For example, students claimed that they could then better follow during the lecture, or that they could get a better understanding of the content during the lecture (see Table 4). These two aspects were mentioned so often (by at least 13% of participants) that they were assigned to own categories as reasons for preferring the flipped lectures (categories 2 and 3). Regarding the activities for successful learning from lectures according to Lew et al. (2016), being able to think and follow during the lecture, and to already gain an understanding of the content, yields a benefit for activity 2, namely 'Processing of the information presented during the lecture'.
Another advantage of the flipped lectures mentioned by at least 4% of students was that they did not only have to concentrate on writing (category 4). This can again be beneficial for processing the information presented during the lecture, but also for retaining attention, as the two sample answers in Table 4 indicate.
The final upside mentioned by at least 4% of students (category 5) was that they could cope with unclear points better in the case of the flipped lectures. The answers were again mostly related to the processing of the instructors' explanations during the lecture, but were sometimes also related to the pre-class processing of the script (examples see Table 4).
Overall, the responses to the open question to state a reason for the preferred lecture type indicate that in the flipped lectures the students could in particular better process the information discussed during the lecture (and maybe also gain an understanding of the content). Two important reasons for this became evident: students already had an idea about the content when coming into class and they did not have to concentrate on writing during the lecture.
Reasons for preferring the traditional lecture: Of the 31 students who preferred traditional lectures, 30 stated a reason for their preference. The analysis of these reasons led to six categories that each included at least 4% of students. These categories and the proportion of students referring to them in their answers can be seen in Table 5.
The category most often mentioned was that attention or concentration was greater in the traditional lectures or lower in the flipped lectures, which directly refers to the activity 'paying attention' of our framework for the questionnaire construction. An explanation might be that if a topic covered in the flipped lecture had already been understood before class by a student, it might have been harder for him/her to maintain attention if the same topic was discussed in class again. Table 4. Reasons for preferring the flipped lectures that were mentioned by at least 4% of the students; an answer could be assigned to multiple categories.

Category
Description Example Per cent (N = 98) 1. Pre-processing of the content The students perceived as an advantage of the flipped lectures that a pre-processing of the content already took place before class.
(1) 'You already have ideas about the lecture content in advance, which leads to a better understanding.' (2) 'You can view the lecture content already in advance, and can better think during the lecture.' 27.6% 2. Thinking and following during the lecture The students perceived as an advantage of the flipped lectures that they could better follow the instructor and think about the content already during the lecture.
(1) 'Since you already have an idea about the content, you can better concentrate and think instead of writing like a robot.' (2) 'If you have time to prepare, you can better follow during the lecture.'

18.4%
3. Better understanding during the lecture The students stated as an advantage of the flipped lectures that they could already gain an understanding during the lecture.
'The possibility to prepare for the lecture leads to more understanding during the lecture.' 13.3%

Focus not on writing
The students emphasized positively that in the flipped lectures the focus lay not only on writing/copying from the board.
(1) 'You do not only have to concentrate on copying from the board. Instead, you can follow the lecturer's explanations.' (2) 'I pay attention instead of taking notes.' 8.2%

Opportunities for coping with unclear points/gaps
The students stated that the flipped lectures made it easier for them to cope with unclear points/gaps in understanding.
(1) 'In the flipped lecture I could better concentrate on the content during the lecture, and, if necessary, already remedy unclear points.' (2) 'You have more time/possibilities to prepare the lecture in advance, and can already formulate questions concerning unclear points.' 4.1% 6. Other benefit of the flipped lecture The students stated another benefit of the flipped lectures. 'The possibility to discuss the topics with the instructor once a week is very helpful.' 13.3% Table 5. Reasons for preferring the traditional lecture that were mentioned by at least 4% of students; an answer could be assigned to multiple categories.

Category Description Example
Per cent (N = 98)

Higher attention/Better concentration
The students stated that they were more attentive or more concentrated in the traditional lectures.
(1) 'I personally can concentrate myself better.' (2) 'I get distracted more easily on Mondays.' (the flipped lecture sessions took place on Mondays) 6.1% 2. Better structure The students stated that the traditional lectures were better structured.
'The traditional lecture is much better structured.' 5.1%

Preference for delivery of new knowledge
The students stated that they prefer to get new knowledge delivered (first), and work on the topic for themselves later.
(1) 'I personally prefer to get new content delivered.' (2) 'First receiving explanations, then working independently. This way I learn better.' 5.1%

Completeness
The students perceived the traditional lectures as complete (or the flipped lectures as incomplete).
'I like the flipped lecture, but unfortunately not all themes in the script are discussed.' 5.1%

Less time effort
The students stated that the traditional lectures were more time-efficient or that the flipped lectures required too much time (at home or during the lecture).
(1) 'The time effort at home is minimal, and the focus can lie more on tasks than on the script/preparation/ post-class processing.' (2) 'The flipped lecture takes away time as it is based on students' activities.' 4.9%

Other benefit of the traditional lecture
The students stated another benefit of the traditional lectures. 'The post-class processing on Wednesdays is easier, and I feel more certain.' (Wednesdays refers to the traditional session)

10.2%
A second advantage of the traditional lecture mentioned by at least 4% of students was the structure (category 2). A clear structure of the lecture is particularly perceived as beneficial by students for their note-taking (Van Meter et al., 1994). In the case of traditional advanced mathematics lectures, the traditional DTP (definition, theorem, proof) format provides a clear (local) structure of the content.
A third perceived advantage of the traditional lecture mentioned by at least 4% of students was the matter 'completeness' (category 4). The traditional lectures, in which all formal content was written on the board, were viewed as complete, while the flipped lecture sessions were sometimes seen as incomplete because not all issues of the script were discussed.
Finally, Table 5 indicates that some students found the traditional lectures more efficient with respect to time. Furthermore, some students just preferred to have information delivered first before engaging with content for themselves (category 5).
Overall, Table 5 illustrates some aspects that could be improved in a future implementation of our lecture-oriented adaptation of the flipped classroom. However, the share of participants mentioning each of these criticisms was relatively low. In particular, although a time effort was necessary for the preparation of the flipped lectures, only a few students mentioned the time effort as a reason for preferring the traditional lectures.

Results concerning the influence of the flipped lectures on students' post-class processing of a traditional lecture
First, we asked students if the in-class activities of the flipped lectures had influenced their post-class processing of a traditional lecture (see Section 5.1). 31.6% selected the answer option 'Yes', 63.3% selected 'No', and 5.1% did not give an answer.
For those who selected 'Yes', we then asked, in an open question, how these activities influenced their post-class processing of a traditional lecture. All of these students specified this influence here. The categorization of the answers based on the data led to five categories that each included at least 4% of students. These are shown in Tables 6 and 7.
The main aim of the in-class activities was to illustrate to the students how they could gain an understanding of formal definitions and formal proofs. Corresponding activities were, for example, discussing examples and visualizations for concepts given by a formal definition, discussing warrants in the steps of proofs, or reflecting on the ideas behind the proofs (see Section 3). Table 6 shows that these activities were adopted by some students for their post-class processing of a traditional lecture. Furthermore, some students stated that they have gained meta-knowledge of how they could process a traditional lecture at home after class (category 1 in Table 6). However, Table 7 indicates that this proportion was relatively small. Possible reasons will be discussed in Section 6.2. Furthermore, 4.1% of the course participants stated that the activities of the flipped lectures had even led to less post-class processing of the lecture (category 4). This is not the direction of influence of the flipped lectures that we had intended (all other categories point out a change in direction towards a more-intensive or at least more deliberate post-class processing). One explanation for these answers might be that these students did not describe an influence of the activities of the flipped lectures on their post-class processing of a traditional lecture (as asked in the question), but on their post-class processing of the flipped lectures, in which the post-class processing activities already took place in class. Since the questionnaire was anonymous, we could unfortunately not ask these students further.

Summary of the results
As mentioned in the literature review in Section 2, there are several problems concerning students' learning in traditional advanced mathematics lectures. In particular, they are Table 6. Answer categories to the question of how the activities of the flipped lectures influenced students' post-class processing of a traditional lecture; an answer could be assigned to multiple categories.

Category
Description Example 1. Better post-class processing of the lecture The students stated that they are now able to better process their notes/rework through the lecture after class.
(1) 'It helped me understand what post-class processing of the lecture means.' (2) 'If I have time during the week, I at least know how to carefully post-class process a lecture, for example by constructing similar proofs, proving yourself, and thinking how each step in a proof is deduced.' (3) 'Quicker post-class processing and learning.' 2. More intensive processing of proofs The students mentioned activities indicating that they now process proofs presented in the lecture at home more intensively.
(1) 'More critical look at proofs. Maybe search for different methods of the proof.' (2) 'I have more than usually tried to make sense of every step of the proofs, which helped me a lot for solving the homework problems.' (3) 'It is helpful to visualize the idea behind a proof.'

Use of examples/ visualizations
The students stated that they now use examples or visualizations to make sense of the lecture content.
(1) 'You now know how you can better postclass process a lecture, for example with the use of visualizations.' (2) 'More detailed processing of the content.
Search for examples.'

Less post-class processing
The students stated that they now post-class process the lecture less.
'I spent less time for post-class processing.'

Other influence
The students stated another influence of the flipped lectures on their post-class processing of a traditional lecture.
(1) 'I searched for answers to the questions that had been discussed in class.' (2) 'I use more flashcards than I used to.' (3) 'I have never reworked through the lecture after class before.' often busy copying definitions, theorems, and proofs from the board and cannot be attentive towards the instructor's explanations. This might make it hard for them to gain an understanding of the content during the lecture, so that an intensive post-class processing is necessary. However, students often do not process lectures intensively. But even if they invest a considerable amount of time for post-class-processing, making sense of the formal content (the definitions, theorems, and proofs) might be difficult for them because often nobody illustrates them how to do this. Furthermore, students often do not note the lecturer's oral comments, which are essential for making sense of the formal content.
To address these problems, we implemented a lecture-oriented flipped classroom, in which the instructor used the in-class time to illustrate how she would try to make sense of the definitions, theorems, and proofs the students had become acquainted with before class (provided in a script). This adaptation of the flipped classroom aimed at combining advantages of the flipped classroom and traditional mathematics lectures held by an expert in mathematics, who can communicate the structure and core ideas of that subject and can model the mathematical behaviour. Since in a flipped classroom students' contact with the definitions, theorems, and proofs takes place before class, they no longer have to copy these in class, but can instead concentrate on the lecturer's explanatory remarks, and might also be more likely to include at least some of these in their notes. Furthermore, since activities for making sense of the content now take place in class, the problem that many students do not post-class process lectures intensively can be addressed. Finally, instructors' illustrations of how mathematical experts make sense of definitions, theorems, and proofs can provide students with meta-knowledge of how they could post-class process a traditional mathematics lecture.
Our subsequent empirical study in a proof-oriented analysis course with one flipped and one traditional lecture per week indicated that some of these possible benefits just mentioned did actually occur. First, students' answers to the Likert items investigating the extent, to which they (could) perform activities that are important for successful learning from lectures according to Lew et al. (2016) (see Section 4), suggested that in the flipped lectures the students could better process the information presented, which might have helped them to better make sense of the formal content during the lecture. Second, students were more likely to note down additional comments, which is important for making sense of the content on the basis of the notes taken later again, for example when preparing for the exam. In addition, the data showed that students omitted important activities for gaining an understanding of definitions, theorems, and proofs in their post-class processing of the traditional lectures like filling gaps in the proofs covered. Hence, incorporating such activities into the flipped sessions might have further helped students to make sense of the definitions, theorems, and proofs covered. And finally, since the students invested more time out of class with the content of the flipped sessions, they probably engaged with the content of the flipped lectures more.
Nevertheless, some of the potentialities of the lecture-oriented adaptation of the flipped classroom hypothesized in Section 3.2 have not been fully realized in our implementation. The students have not been more attentive in the flipped lectures. Hence, the format did not have the same effect on students' attention as it was shown for the flipped classroom in other implementations (see for example Jungić et al. (2015)). Furthermore, some students perceived the flipped lectures as less structured, or even incomplete due to the necessity to focus just on certain content in class. Finally, also the special benefit of our lecture-oriented adaptation of the flipped classroom -that is, to provide students with meta-knowledge of how they could post-class process a traditional mathematics lecture and with corresponding strategies they might also adopt -was only realized for a minority. Even though some participants reported that they had gained such knowledge and had adopted some of the in-class activities for their own post-class processing, 63.3% claimed that the in-class activities of the flipped lectures did not influence their post-class processing of a traditional lecture.

Discussion of the results and possible implications for teaching
Our results suggest several important implications for teaching. First, the data indicate that the lecture-oriented flipped classroom we implemented was useful in addressing some problems of traditional mathematics lectures while still keeping the lecture format as the mode of teaching. In particular, it could address the problem that in traditional mathematics lectures students often cannot process the lecturer's explanations as they are too busy copying definitions, theorems, and proofs from the board, and the problem that students often do not write such explanations down. This might have helped many students make sense of the definitions, theorems, and proofs covered (although the data presented here cannot prove the latter assumption).
However, other possible benefits have not yet been fully realized. Students' attention was not greater in the flipped lectures. A reason might be that the flipped lectures were also teacher-centered (although many activities took place in the form of a dialogue between instructor and students). A closer involvement of students in the activities illustrating how students might make sense of the definitions, theorems, and proofs during the flipped lectures could help to address this problem. For instance, some of the activities from our implementation (see Section 3.3) could also have taken place as a discussion between students, for example debating the warrants in the steps of a proof, or a discussion of examples and counterexamples that either fulfil a definition or not.
The second important benefit that has only been realized for a minority was the induction of a change in the students' study behaviour concerning their post-class processing of traditional mathematics lectures on the basis of the behaviour modelled by the instructor during the flipped lectures, although the instructor emphasized the importance of the corresponding activities for making sense of new definitions, theorems, and proofs. Hence, the provision of meta-knowledge of how students could post-class process mathematics lectures did not bring about a change in their study behaviour for many students. This phenomenon is also documented in the literature: many students do not change their learning behaviour even if they might benefit from a change (Dembo & Seli, 2004;Jakubowski & Dembo, 2004). Important reasons are that they do not want to change or do not know how to change (Dembo & Seli, 2004). With respect to our course, for example, students might not have seen the immediate benefits of performing the in-class activities for themselves after the traditional sessions (the exam lay far away). A solution might be to illustrate the benefits of the activities directly with the weekly homework problems, for example by letting students experience failure to solve the problems without these activities (or experience that they would need a long time to find a solution), but that they would be able to solve these after such activities. The second reason suggested by Dembo and Selithat students do not know how to change -could be addressed by explicitly practising with students the activities illustrated during the flipped lectures for the post-class processing of a traditional mathematics lecture. Afterwards, it might also be important to incorporate activities that foster students' maintenance of their newly acquired post-class processing strategies throughout the course (Jakubowski & Dembo, 2004). This might raise the proportion of students really adopting the activities of the flipped lectures for their own post-class processing of mathematics lectures.
We should also acknowledge the challenge that the flipped classroom requires an investment of time by the students. As Figure 8 suggests, most of our participants did actually invest time in preparing for the flipped sessions. However, one needs to think about possibilities if they had not. One example might be to encourage preparation by an accompanying task that is also graded.

Limitations and outlook
Finally, we mention some limitations of our research, which may be starting points for future research.
The most important limitation is that our results are only based on self-reports. It is, for example, unclear whether students really processed the information in the flipped lectures more effectively or if their note-taking actually differed. More objective measures would be necessary here. In particular, the study cannot show if students really gained a deeper understanding of the lecture content, although the proportion of students who passed one of the two final exams after the semester was, at 73.1%, higher than in some other years. In the following year, for example, in which the whole course was held traditionally again, only 62.0% passed although that exam contained more calculation and less proof tasks. However, this cannot show that this difference was due to the flipped classroom because the instructor was different, and we do not know if the students in the following year might just have had lower mathematical entry qualifications. Whether students actually gain a better understanding in such a lecture-oriented flipped classroom would therefore be an important issue for future research.
Second, our study was carried out only in one particular advanced mathematics course. Further investigations in the context of other courses are therefore necessary to find out the extent to which our findings regarding the flipped classroom in the lecture-oriented adaptation we implemented are generalizable.
A last point for future research might be to implement a teaching scenario, such that more of the benefits mentioned in Section 3.2 are realized for the majority of students. Hereby, it might be especially valuable to adapt the lecture-oriented flipped classroom in a way that the in-class activities during the flipped sessions really induce a change in the students' study behaviour concerning their post-class processing of mathematics lectures. If this succeeds, the format might not only support students' learning within the course in which it is used, but might also be helpful for their learning in other advanced mathematics lectures held traditionally.

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
We acknowledge support by the Open Access Publication Fund of Humboldt-Universität zu Berlin.