Development of educational leaders’ adaptive expertise in a professional development programme

ABSTRACT This study considers the extent to which a professional development programme for educational leaders in a research-intensive university contributes to participants’ adaptive expertise in the domain of leading educational change. We evaluated the programme by asking participants to execute an authentic task at the beginning and end of the programme and compared the outcomes with participants’ self-reported learning gains. While participants report they have substantially learned from participating, according to the task scores there is no significant progress in the development of adaptive expertise. Suggestions are offered to include more purposeful practice and more reflective activities in the programme.


Introduction
Enhancement of the educational provision of institutions for higher education is needed if students are to be prepared for a fast-changing world (Fung et al., 2017). In order to achieve such improvements, universities are, among others, developing new and enhanced educational strategies, working towards curriculum change and innovation, introducing technology for delivering the curriculum (Marshall, 2018), and building learning spaces and other facilities to support student learning (Bonem et al., 2020;Clinton & Wilson, 2019). Typically, achieving educational changes such as these is considered to be a complex process involving many different groups of people, substantial budgets, and long periods of time, and since the cooperation of busy academics is needed to effect these various changes, leadership is essential (Fullan, 2002).
The task of achieving necessary changes is assigned to leaders at all management levels of the university organisation (Blackmore & Kandiko, 2012, p. 136;Bolden, 2011;Bolden et al., 2009). Midlevel leaders with a responsibility for education, such as associate deans (Floyd & Preston, 2018), heads of studies, or programme directors (Milburn, 2010), may have the most direct influence on curriculum change. Their tasks and responsibilities differ across universities, but they often lack formal power (Floyd & Preston, 2018), which makes their role in leading educational change processes more complex and challenging (Floyd & Preston, 2018;Milburn, 2010;Preston & Floyd, 2016;Vilkinas & Ladyshewsky, 2012).
Fung and colleagues found that universities have begun to offer opportunities for academic staff to develop expertise for leading educational change (Fung et al., 2017). The current study evaluates one programme's contribution to the development of adaptive expertise of midlevel educational leaders.

Adaptive expertise for leading educational change
Expertise research focuses on what constitutes expert performance in various domains and individual differences in how people acquire expertise. Among the main findings is that experience and deliberate practice are most important for achieving high levels of reproducible performance and continued improvement (Ericsson, 2006(Ericsson, , 2009. However, Ericsson (2014, p. 184) referred to research outcomes showing that there is 'often not a significant correlation between the amount of experience or professional training, and performance'. According to Ericsson, deliberate practice is necessary to achieve an expert level, and he defines deliberate practice as practice with the intention of improving performance and perfecting specific skills, guided by a mentor or teacher (Ericsson, 2014). This topic has been well researched for domains such as music, chess, sports, and air traffic control (Ericsson, 2009;Ericsson et al., 2006Ericsson et al., , 1993. What these domains share is a relatively stable context with predictable tasks, which does not apply to the domain of educational leadership. Leaders of educational change are professionals who need to manage novel problems daily in a complex higher education environment. It is often their responsibility to solve planning problems, contribute to university or faculty policies in areas of education and educational change, or develop and implement solutions for educational problems, both large and small, that are likely to be different each time. To perform well, these professionals need to be able to adapt to changing circumstances and demands, and they need what Hatano and Inagaki (1986) referred to as 'adaptive expertise', which the authors distinguished from 'routine expertise'. Individuals with high levels of both routine and adaptive expertise exhibit high levels of task performance. The difference between routine and adaptive expertise becomes clear once individuals are confronted with an unfamiliar situation: 'while individuals with routine expertise struggle with the new demands, adaptive expertise allows for easily overcoming the novelty and quickly regaining a high level of performance' (Bohle Carbonell et al., 2014, p. 15).

Professional development for adaptive expertise
This study concerns the development of adaptive expertise in midlevel educational leaders in a research-intensive university as a goal of a professional development programme. Some literature suggests that any professional development aimed at enhancing adaptive expertise should provide participants with: (1) the opportunity to acquire knowledge and skills relevant to the domain (Bohle Carbonell et al., 2014), (2) the opportunity to gain experience with dealing with change and novel tasks (Bohle Carbonell et al., 2014), and (3) multiple opportunities for deliberate practice in the domain (Ericsson, 2006). The first characteristic addresses the domain knowledge that every routine and adaptive expert should have; the second adds training for adaptive expertise by focusing on change and novel tasks; and the third characteristic, deliberate practice, is regarded as an essential element to achieve expert performance (Ericsson, 2006). However, Ericsson (2014, p. 194) recognised that it is difficult to design deliberate practice for professionals -in other words, to find a series of novel tasks that intentionally increase in difficulty. He also noted that professionals often have limited opportunities to receive feedback and time to practice (p. 191). In a review study (Bohle Carbonell et al., 2014), and a meta-analysis (Macnamara et al., 2014), no substantial support was found for deliberate practice, which is essential for developing adaptive expertise. However, Ericsson and Harwell (2019) observed that deliberate practice was originally conceptualised to describe how (individual) musicians develop their expertise when coached by a mentor, and that in other professions this type of deliberate practice does not exist and might be better referred to as 'purposeful' practice, meaning deliberate practice without the support of a mentor. Often, only 'naïve' practice is possible in professions, which consists simply of executing the job without the expectation of intentionally developing performance (Ericsson & Harwell, 2019, p. 5).
Ericsson as well as Macnamara and colleagues agree that other factors also are important for explaining individual differences in professionals' expertise (Ericsson, 2016;Macnamara et al., 2016), both in the work environment and in training design. In the work environment, it is important to have a supportive climate in which supervisors encourage professionals to develop their domain knowledge and allow them to make errors (Bohle Carbonell et al., 2014;Hatano & Inagaki, 1986). Work as well as professional development programmes could offer a variety of tasks to allow professionals to gain experience of dealing with change. Such a variety of tasks stimulates the flexible usage of domain knowledge, discovery of commonalities, and description of problems regarding deep structures rather than surface details (Barnett & Koslowski, 2002, p. 261, summarising earlier research). Ward et al. (2018, pp. 43-46) also stressed the need for a variety of training tasks that allow practicing required skills for solving complex problems and noted that another important factor is the opportunity for feedback and reflection. Because professionals often lack performance feedback on a daily basis (Ericsson, 2014), they have to personally monitor and reflect on their progress, choose suitable next steps (Van Gog et al., 2005), and develop their own solution strategies for authentic tasks (Bohle Carbonell et al., 2014, p. 26). Researchers have found that participating in a learning community could be useful when the community encourages reflection on experiences and errors (Wetzel et al., 2015); such a community could be part of a work situation or group-based professional development programme.
In this study, we evaluated the extent to which a professional development programme for groups of midlevel educational leaders, focused on leading educational change, contributed to participants' adaptive expertise in the area of curriculum design and planning of educational change. The programme is described in the next section.

A professional development programme at Utrecht University
A programme was designed at Utrecht University that focused on leading educational change for senior academics in midlevel educational leadership roles. The programme aims to support participants in building knowledge of and experience with challenging change processes that lead to improving the quality of curriculum and learning environments in the ever-changing context of higher education.
The programme includes eight residential 24-hour meetings over a period of 15 months. Topics in the areas of higher education pedagogy and leading change processes in universities are introduced by guest lecturers. Ample literature connected to these subjects is provided, and a week-long study tour to foreign universities is also arranged. A substantial aspect of the programme is that each participant leads a complex, novel-to-them educational change project in their own department or faculty, which provides them opportunities to test new knowledge and ideas from the programme. Topics are chosen through interacting with programme leaders and participants, and provide opportunities for connecting theory with practicing on a just-in-time basis. In this way, the programme leaders play a mentoring role for the whole group, which thus functions as an informal learning community. Multiple opportunities for reflection and feedback are provided throughout the programme, such as through group peer review and discussions of the projects. A more extensive description of the programme can be found in Grunefeld et al. (2015). Compared to other programmes for midlevel educational leaders, the programme is unusual in terms of its duration and its focus on leading educational change rather than management (Grunefeld et al., 2017).

Focus of this study
The current study focused on adaptive expertise in curriculum design and planning for a successful change process in a university. Knowledge in these areas is essential for midlevel leaders working towards improving the quality of education or developing a new programme (Scott et al., 2008).
The research question was as follows: To what extent does this specific professional development programme for midlevel educational leaders in a research-intensive university contribute to participants' adaptive expertise in the domain of curriculum design and planning of educational change? The research question was divided into four subquestions, with two addressing the difference in expertise before and after participation, and the remaining two the characteristics of the programme and work context that might be conducive to developing adaptive expertise: (1) What was the change in adaptive expertise between the beginning and end of the programme? (2) How did participants perceive the change in their knowledge between the beginning and end of the programme? (3) Which knowledge addressed in the programme was used by participants in completing the task? (4) Did participants experience a work climate and task variety in educational change that can be viewed as conducive to developing adaptive expertise?

Methods
A one-group pre-test post-test design was used. This test design utilised a mixed methodology with three instruments, including both quantitative and qualitative measures. The investigation included educational leaders in four cohorts of the educational leadership programme, conducted over a period of three years. In total, 57 (of 66) educational leaders participated in a pre-test, 30 of whom also participated in a posttest, resulting in a post-test response rate of 53%. Planning and time investment were reasons for non-participation. We used the MEDEC instrument (Grunefeld, 2020) before and after the programme for the first subquestion. This instrument consists of three parts: an authentic and representative task to capture adaptive expertise (Ericsson, 2014) in the domain 'curriculum design and planning of educational change', a rubric, and a scoring procedure. The task was to design new curriculum and project plans in the role of an informal educational leader. This required a degree of adaptive expertise, since it concerned an unexpected, challenging, and novel-to-them task (Ward et al., 2018), and participants needed to apply knowledge of the domain. The rubric focused on applying domain knowledge of curriculum design using the concept of constructive alignment (Biggs & Tang, 2011;Tyler, 1949Tyler, /2013, and on applying knowledge of educational change in the project plan using the success factors identified by Havelock and Huberman (1977) and Gibbs et al. (2008). Constructive alignment was detailed using five criteria, and the success factors for educational change using six. The scores (scale 0-10) on the 11 criteria represented the extent to which domain knowledge was used in the areas of curriculum design and educational change process design. The validity of this instrument was demonstrated for all criteria together and for two of the curriculum plan criteria and four of the project plan criteria (Grunefeld, 2020). We chose to use all 11 criteria in this study. Differences between the pre-and post-tests were analysed using a repeated-measures MANOVA approach. Self-reported data were gathered for answering the second and fourth subquestions by means of the second instrument, which consisted of two short online questionnaires. After performing the task, participants were asked to retrospectively grade their knowledge before and after participating in the programme on six areas derived from the goals of the programme. Differences were analysed using a repeated-measures MANOVA approach. Questions were asked about aspects of the work climate, such as supervisor support, and about participants' experience with educational change by asking about time spent on educational innovation projects. Later, in an additional online questionnaire, questions were asked about participants' educational background and leadership role.
To answer the third subquestion, the third instrument used was a semi-structured interview that was conducted with each participant after completing the MEDEC task for the second time. All 30 participants were interviewed; however, four interviews were excluded due to recording problems. Participants were asked to reflect on their: approach to solving the task, design steps and reasoning, and use of knowledge acquired in the programme. The interviews were coded with a focus on three topics in the area of domain knowledge and skills: the extent to which any theory and models were used in designing the curriculum and project plan, whether and which examples were used, and whether participants remembered how they had performed this task a year earlier in the pre-test. In addition, all references in the interview to participating in the programme or other learning in the past year were coded. Coding was performed by the first author. An audit trail (Akkerman et al., 2008) was conducted by the second author to assess the acceptability (trustworthiness) of the coding; a description of the audit process and the final auditor report is available on request. Using this approach, a quantitative measurement (MEDEC) was complemented by two qualitative measurements (questionnaires and interview).

Subquestion 1, Changes in expertise measured with the MEDEC instrument
The scores on the 11 criteria of the MEDEC instrument can be found in Table 1. The scores of the post-test were not significantly different from the equivalent scores on the pre-test. A repeated-measures MANOVA test with all 11 criteria resulted in Pillai's Trace V = 0.43, F (11, 19) = 1.311, p = .291, while the partial eta squared was 0.43.

Subquestion 2, Changes in knowledge as perceived by participants
In their answers on the online questionnaire immediately after performing the post-test, participants estimated that their expertise at the end of the programme, operationalised as level of knowledge, had improved significantly since the start for all six areas of knowledge (see Table 2).
A repeated-measures MANOVA resulted in Pillai's Trace V = 0.86, F (6, 24) = 25.281, p = 0.000, the partial eta squared was 0.86. All univariate differences were also significant (tested with Bonferroni correction). In addition, two more questions were asked about Scale 0-10, 0 = no information, 10 = rich information about this criterium. participants' estimation of their level of expertise in designing a new curriculum and writing a project plan for an educational innovation project. The scores followed a similar pattern, with respectively a 6.2 and 6.1 before the programme, and a 7.8 and 7.6 after the programme.

Subquestion 3, Which knowledge addressed in the programme was used?
The leading question for analysing the interviews was: 'Which knowledge and examples that were addressed in the programme did participants use in the task?'. Most interviewees (20 out of 26) had used concepts and examples that were addressed during the programme. They used relevant concepts such as 'stakeholder analysis' while working on their project plan (13 times), and 10 interviewees used constructive alignment or related concepts while working on the curriculum plan. Examples of other topics to which participants referred include using a hybrid teaching and learning model; the concepts of constructive and destructive friction; the strategic value of stakeholder analysis and the importance of having conversations with all people involved; and that it is acceptable to begin with a project plan, as ideas for the content of a curriculum can arise later. Only four (out of 26) interviewees mentioned that they had not considered specific examples to help them with the task, while the others used examples from their own experience or from other participants in the programme, or thought of examples encountered during the study tour or from guest lecturers or books.
Interviewees were asked whether they remembered the design they developed in the pre-test and whether they approached the task differently in the post-test. Fifteen interviewees (of 26) did not remember the previous task, and one of them reflected: 'I see now that, although the programme introduces several models, when confronted with a complex request, it's easy to fall back on knowledge and experiences from before participating in the programme'. The comments of the other participants were ordered into three categories: five interviewees said they had a better structure now compared with the pre-test, five had a better idea of how an effective curriculum should be designed, and five thought they had a better idea of the process leading towards a new curriculum (see Table 3 for some examples of their statements that show the perceived difference between completing the pre-test and post-test tasks).
In addition, we selected the participants who had made the most substantial progress, based on the pre-test and post-test scores of the MEDEC instrument. In order to find them, we calculated the total scores for the pre-and post-tests for each participant (the Table 2. Participants retrospectively estimated knowledge before and after the programme. sum of the 11 criteria). We then ranked the participants based on the difference between the pre-and post-tests. Fifteen participants made progress with on average nine points more on the post-test, while nine participants had on average 13 points less on the posttest than on the pre-test. The three participants with the greatest positive difference had post-test scores that were around 22 points higher than the pre-test. These three participants were aware that they approached the task in the post-test differently from the pre-test, and they said that they now better understood the design process. All three remembered and used important theory, methods, or examples from other participants or from the study tour. For example, stakeholder analysis was mentioned by all three and constructive alignment by two of them, and two mentioned that they now better understand the context and organisation of the programme and have a different stance on designing education. These results show that not only had their scores on the MEDEC instrument substantially improved, but also that the programme had made a difference in their domain knowledge and expertise.

Subquestion 4, Other factors related to development of adaptive expertise
The prevalence of two conditions that might have influenced the development of adaptive expertise was investigated. Regarding work climate, participants rated the support by supervisors for normal tasks at 6.7 on a scale of 0-10 (see Table 4). We thus regard the work climate on average as supportive, which might have facilitated expertise development. There is a significant positive Pearson correlation (r = .44, p = .037), showing that supervisors who are rated as supportive for normal tasks are also rated as supportive for a project in which they were not involved.
Secondly, in the past year, these leaders spent on average a third of a full-time week load (between 0.03 fte and 1 fte; M = 0.32 fte, SD = 0.28) on a variety of educational innovation questions and projects, where the mean number of projects was 15.6 Table 3. Examples of statements showing perceived difference between pre-test and post-test.
Statements expressed just after the post-test Structure 'This time, I started with the stakeholder analysis, and I see now that at this stage in the process, it is the design phase, and planning for the implementation phase is hardly necessary at this stage'. 'Then, my approach was much more unfocused'. Curriculum 'Then, I thought more of how to convince the senior leadership; now I thought more of the content of the programme'. 'Then, I had no idea of balance in a curriculum'. Process 'Then, I had no idea of what the process towards a new programme would be'. 'Now, I'm less concerned with bureaucratic issues, I am thinking more about engaging people in the process'. Table 4. Work climate (scale 1-10, 1 = no support, 10 = excellent support).
M SD Min Max How do you rate the support of your supervisor regarding your normal tasks? 6.7 2.09 1 9 How do you rate the support of your supervisor regarding your innovative project? (NB. Only answer this question if your supervisor and your client are different persons) 4.8 3.08 1 9 How do you rate the support (approachable, encourages people, organises help) of the client of your innovative project?
7.3 1.35 4 10 (SD = 15.9). We regard this level of variety of tasks as presenting sufficient opportunity to gain experience and practice required skills.

Conclusions and discussion
The research question was: 'To what extent does this professional development programme for midlevel educational leaders in a research-intensive university contribute to participants' development of adaptive expertise in the area of curriculum design and planning of educational change?' We investigated this question in two ways. We asked participants to rate their level of knowledge before and after the programme, and they estimated that they had made significant improvement in all areas. This self-report measure was complemented by the MEDEC instrument, to provide a less biased measurement. The MEDEC instrument detected no significant differences in adaptive expertise. Possible explanations for this unexpected contrast are discussed next. A first explanation for the contrast, more specifically for the limited progress on the MEDEC instrument, could be that participants did develop adaptive expertise, but that this was not revealed by the MEDEC instrument. It might be that the task, although deemed relevant, was not representative for the work package of educational leaders. However, we do not consider this a plausible explanation, because the task was carefully selected and captures an essential part of the domain. Factors in the workplace that are conducive to expertise development (work climate and variety of tasks) appeared to be at an acceptable level, and earlier evaluative research showed that the programme contributed to forming a learning community (Grunefeld et al., 2015). We therefore do not consider these factors to account for the lack of progress on the MEDEC instrument.
A second explanation for this unexpected difference could be a halo effect (Feeley, 2002) of participants' enthusiasm for the programme as a whole (Grunefeld et al., 2015), and a third, as was indicated by a comment of one of the participants, that participants, at the moment of performing the task, did not have the appropriate domain knowledge available. If this third explanation is valid, it suggests that new domain knowledge had not been learned well enough to function and that adaptive expertise had not been developed sufficiently.
This leads to a fourth explanation, that the programme failed to support the development of adaptive expertise by not providing sufficient opportunity for deliberate practice to improve individual performance in tasks in the domain of curriculum design and planning of educational change. The practice organised in the programme could probably be classified as structured practice, which according to Ericsson and Harwell (2019) consists of group activities designed by a coach or teacher, which are not tailored to enhance participant's individual level of adaptive expertise. Assuming this explanation is true, what advice can be found in the literature to improve the programme? As Ericsson (2014) indicated, what matters is not only the amount of deliberate practice but also the specificity of the training tasks. One way to improve the programme may be implementing case-based learning (Mumford et al., 2009), which should encourage participants to consider and develop mental models for leadership problems and solutions. This should be performed with sufficient intensity, as Mumford and colleagues observed that in most leadership training programmes at that time, case-based learning lacked depth of analysis, or the number and quality of the cases, was not sufficient to cover the domain. Another suggestion is to increase the depth of reflection by using, for example, a 'reflection prompt protocol', as proposed by Wetzel et al. (2015), to train adaptive expertise by inviting critical dialogue about cases and experiences. A further option was proposed by Ward et al. (2018), who presented a model for leadership expertise that might be useful for the professional development of education leaders. The model describes the 'mental modelling processes' that leaders engage in when dealing with unexpected and challenging situations. Ward et al. (2018) describe detailed principles for training these processes, that would need interpretation when applied to the programme under investigation. The above suggestions to improve the programme by increasing the number of cases to be analysed, and by performing deeper reflection on cases and experience, would increase the time-ontask and thereby probably the resulting learning outcomes (Chickering & Gamson, 1987). The duration of the programme could alternatively be extended, and existing forms of practice could be strengthened, for example, through continued reflective activities such as action learning sets, workshops, or mentoring.
The outcomes of this study may call for a redesign of the programme and other comparable professional development programmes for leaders in higher education, to improve the development of adaptive expertise. However, because this programme is already seen as a successful programme (Grunefeld et al., 2015), such redesign might be interpreted as aiming at 'teaching to the test', and thereby limit opportunities for learner control and realisation of other desirable outcomes of a programme. Such redesign should in any case consider all of the desirable outcomes of the programme as well as reliable and valid methods of measuring the achievement of these outcomes.