Practical independent research projects in science: a synthesis and evaluation of the evidence of impact on high school students

ABSTRACT Practical independent research projects (IRPs) are a feature of school science in a number of countries. To assess the impact of IRPs on students, a systematic review of the literature was undertaken. Thirty-nine papers met the review inclusion criteria, reporting on work from twelve countries. The review indicates that IRPs are often associated with wider initiatives such as authentic science, problem-based learning, and project-based learning. There is considerable variability in the nature of IRP work in relation to focus, models of provision, assessment, the involvement of external partners such as universities and employers, and funding, and this diversity affects judgements on the quality of the evidence base on impact. The majority of the research reviewed explored areas such as conceptual understanding, motivation to study science once it is no longer compulsory and attitudes to science, and the development of practical skills. Benefits were identified in relation to the learning of science ideas, affective responses to science, views of pursuing careers involving science, and development of a range of skills. Studies focusing on traditionally under-represented groups indicated that such students felt more positive about science as a result of undertaking IRPs. The review findings indicate that further work is needed to enhance the quality of the available evidence, to consider the ways in which IRPs can be validly assessed, to explore more fully the potential benefits for traditionally under-represented groups, and to explore more fully the potential longer-term benefits of participation in IRPs at high school level.


Introduction and context
This paper presents the findings of a systematic review of the nature and impact of practical independent research projects (IRPs) in high school science, covering their chief characteristics, organisation and assessment, and impact on high school students' learning of science and affective responses to science.
Practical work in school science can be very diverse: at one end of the spectrum is the 'recipe' approach, where a defined list of procedures is followed, while at the other is what can be termed an extended investigation or a practical independent research project (IRP). Such projects can take a wide variety of forms, but share several common characteristics. In essence, they are student-led, open-ended research investigations, often supported by a teacher and/or a university-based or industry-based researcher. Students have considerable control in respect of the question(s) they hope the practical work will answer and the way in which the work is undertaken.
Typically, though not exclusively, IRPs are undertaken by high school students, and, at the upper high school level, the outcome of the investigation is open in that neither the student nor their teacher knows exactly what the investigation will yield. Beyond this, IRPs may involve external sponsorship, and be associated with science competitions, fairs and award schemes. IRPs can also involve a diversity of assessment techniques, including the production of reports and student presentations. Frequently, they take place outside the formal school science curriculum.
For the purposes of this review, IRPs were taken to be student-led, extended openended investigations involving practical work, using Millar's (2004) definition of practical work, i.e. work that encompasses activities involving students in observing or manipulating the objects and materials they are studying.
There appear to be a number of possible reasons for promoting the use of IRPs in school science lessons. First, the notion of 'the students as scientist' is attractive, allowing students to find things out for themselves by pursuing an idea about which they are curious. Second, IRPs are seen as a means of providing students with a realistic taste of scientific research that may motivate them to undertake further study of science. Third, the characteristics of IRPs may be identified in broader, international initiatives of the last twenty years or so, for example 'inquiry-based science', 'problem-based learning' in science and 'authentic science'. These approaches have in common the desire to encourage students to engage in activities where at times they behave like scientists, i.e. their work is authentic in that it follows the approaches scientists take when they are trying to solve problems to which there may as yet be no agreed solution. These approaches are primarily aimed at improving cognitive and procedural outcomes for students, though also aspire to have affective benefits. Roth (1995) argued that for school science activities to be authentic, students need to experience scientific inquiry that has features in common with scientists' activities in that students (1) learn in contexts constituted in part by ill-defined problems; (2) experience uncertainties and ambiguities and the social nature of scientific work and knowledge; (3) learning is predicated on, and driven by, their current knowledge state; (4) experience themselves as parts of communities of inquiry in which knowledge, practices, resources and discourse are shared; (5) in these communities, members can draw on the expertise of more knowledgeable others whether they are peers, advisors or teachers. (p1).
There are similarities between the features of authentic learning described by Roth and the characteristics associated with problem-based learning (PBL). A PBL approach involves students learning through focusing on the investigation, explanation, and resolution of meaningful problems. PBL has its origins in teaching in medical schools, but has now been used in a variety of subjects, including science. Students work collaboratively in small groups, with the teacher acting as a facilitator to guide student learning through the process of solving the problem presented (see, for example, Hmelo-Silver, 2004;Krajcik & Blumenfeld, 2006).
Inquiry-based learning is another related approach that is widely used in the context of science teaching. In a research synthesis of inquiry-based learning in science, Minner, Levy, and Century (2010) identified that one of the ways in which the term is used is to describe a pedagogical approach that teachers employ for designing or using curricula that allow for extended investigations. Drawing on the work of the National Research Council in the USA, Minner et al. (2010) cite the following as the core components of inquiry-based learning for learners: (1) they are engaged by scientifically-oriented questions; (2) they give priority to evidence, which allows them to develop and evaluate explanations that address scientifically-oriented questions; (3) they formulate explanations from evidence to address scientifically-oriented questions; (4) they evaluate their explanations in light of alternative explanations, particularly those reflecting scientific understanding; (5) they communicate and justify their proposed explanations; (6) they design and conduct investigations.
Authentic learning, problem-based learning and inquiry-based learning all emphasise open-ended investigative work, including the sort of work that occurs in IRPs, and hence have been considered in this review.

The aims of the review
The review addressed the following questions: 1. What are the chief characteristics of practical independent research projects (IRPs), including organisation and assessment? 2. What is the quality of the research evidence base on the impact of IRPs? 3. What are the impacts of IRPs on the learning of science and affective responses to science in secondary school students'?
The impetus for the review arose from recent changes in the policy regarding the teaching and assessment of practical work as part of the national examinations for students aged 16 and 18 in England. One outcome of these changes has been a decisive move away from IRPs as part of the core curriculum which, in turn, has led to a concern that there will be a reduction in some of the learning potentially associated with IRPs. While a number of studies have been conducted into the impact of IRPs, to the best of the authors' knowledge, a review of the associated literature has not been published. Thus the review seeks to bring together recent research into the impact of IRPs, in order to synthesise the existing literature and highlight areas suitable for further investigation.

Review methods
The review took the form of a systematic review. Systematic reviews were introduced as a tool in educational research in the early 2000s to synthesise research findings from a range of related studies (see, for example, Gough & Elbourne, 2002;Gough, Oliver, & Thomas, 2012).
The review comprised five main stages: 1. Identification of the literature 2. Extraction of the key information from the literature 3. Production of an overview of the key features of IRP provision 4. Assessment of the quality of the available evidence in the literature on the impacts of IRPs 5. Synthesis of the evidence.

Identification of the literature
The relevant literature was identified through a search was undertaken of the following electronic databases: the British Education Index (BEI), the Education Resources Information Centre (ERIC), PsychINFO and the Social Science Citation Index (SSCI). Additionally, 27 key informants working in the area of IRPs in a range of countries were contacted to identify potentially relevant publications. This was done to add to the robustness of the evidence base, as preliminary electronic searches indicated that there were likely to be a number of publications in the form of grey literature (e.g. reports commissioned by IRP providers) that might not otherwise be identified.
In practice, the identification of the relevant literature posed one of the major challenges for the review as IRP activity can be encompassed by a number of other activities. These include: authentic science, independent and/or extended practical work, inquirybased science, investigative work, practical work, problem-based science and project work. IRP work is also often associated with science competitions and fairs. This diversity in terms necessitated particularly extensive literature searches in order to include all the above terms. Full details may be found in the technical report (Bennett, Dunlop, Knox, Reiss, & Torrance Jenkins, 2016).
The electronic searches identified 1,403 publications, with a further eleven included based on information from the key informants. To identify the literature that focused specifically on the review questions, the following inclusion criteria were developed: 1. One or more of the review questions addressed 2. Focus on students in 11-19 age range 3. Focus on science subjects 4. Date of publication after 2000 5. Students had major input into the question(s) addressed by the IRP 6. Students had major input into the design of the IRP 7. Included practical work 8. Required more than 10 hours of work 9. Entailed production of a report or comparable output 10. Data gathered systematically on students' learning of science and/or affective responses to science 11. Some form of assessment or accreditation 12. Publication written in English.
Criteria 3 and 5-7 were set as they encompassed the definition of IRPs used in this review. The review was limited to work undertaken with students aged 11-19 as this covers the high school period where the majority of IRP work takes place in schools. The search covered the period from 2000 onwards in order to focus on recent work. The criterion of an extended period of time for the IRP work was set to avoid the inclusion of short, even single-lesson, investigations which have become common in some countries. The requirement to produce an output of some form was set to yield information on how IRPs might be assessed. When the inclusion criteria were applied, a total of 39 publications resulted.
Extracting key information from the literature In addition to the wide range of search terms required and the extensive resulting literature, several other factors contributed to the complexity of the review. A number of different research approaches and data collection techniques were used across the studies as a whole, and this research varied in rigour and detail reported. The studies included in the review took place in a range of contexts within and beyond school settings, and in a range of science disciplines. Thus, a particular challenge for extracting key information from papers was the development of a bespoke data extraction instrument to record systematically the wide range of features and considerable variation in practice associated with IRP work and research into its impact. A pilot version was developed by two of the researchers and independently tested on a subset of papers. Minor modifications were made as a result of this to ensure the instrument captured all the essential information. The resulting bespoke data extraction instrument focused on the following information: . Background information (author(s), year of publication, title, source, country of origin, author details) . The aims and research questions of the study . The name of the associated IRP scheme (if applicable) and a short account of the IRP, including: its aims; principal characteristics (e.g. optional or compulsory, duration, organisational details, degree of student control over questions, whether undertaken by individuals or in teams, input from teacher or others, e.g. intern, university researcher); assessment/accreditation arrangements; associated funding . Design of the study and sample details . Data collection techniques (including checks for the reliability and validity of instruments) . Methods of data analysis (including assessment of reliability and validity) . Findings (including any impacts on students' learning, affective responses/attitudes and subject choices or career intentions).

Findings: characteristics of IRPs and overview of provision
The overview is based on 39 publications that report data on impacts of IRPs in enough detail to understand the nature of the IRP and any effects. The publications covered IRP activity in twelve countries, as shown in Table 1.
The majority were from the USA (17 studies) and the UK (8 studies), with two studies coming from each of Australia and Turkey, and single studies from Ireland, Israel, The Netherlands, New Zealand, Qatar, Singapore, Spain and Taiwan. Table 2 summarises five contrasting IRP models to illustrate the diversity in student work undertaken for IRPs and the outcomes reported.
There were three principal contexts in which students engaged in IRPs. In some cases, undertaking IRPs was linked to national policies/agendas. For instance, several USA studies reported on interventions that had secured funding for local initiatives through linking them to policy statements by organisations such as the AAAS (American Association for the Advancement of Science) or the NAS (National Academy of Sciences) (Adams et al., 2009;Dolan, Lally, Brooks, & Tax, 2008;Gibson & Chase, 2002;Sahin, 2013). Secondly, as noted earlier, IRPs were very often associated with wider initiatives, including: authentic science, for instance in Israel (Zion et al., 2004), The Netherlands (Bulte, Westbroek, de Jong, & Pilot, 2006) and the USA (Burgin, Sadler, & Koroly, 2012;Dolan et al., 2008;Rivera Maulucci, Brown, Grey, & Sullivan, 2014); problembased learning, for instance in Qatar (Faris, 2008) and Singapore (Chin & Chia, 2004); and project-based learning, for instance in the USA (Krajcik & Blumenfeld, 2006;Schneider, Blenis, Marx, & Soloway, 2002).
Thirdly, a number of IRP activities were linked to non-governmental groups with a specific interest in promoting IRPs as a way of providing young people with authentic experiences of working as a scientist. Such initiatives typically involved school-university partnerships and included the CREST awards which are run in several countries, including the UK and Australia (British Science Association, 2014; Grant, 2007;Moote, Williams, & Sproule, 2013) and, in the UK, the Nuffield Research Placements scheme (Nuffield Foundation, 2013), The Royal Society Partnerships Grants scheme (Jenkins & Jeavans, 2015) and the Authentic Biology Project funded by the Wellcome Trust (Colthurst et al., 2015;Finegold, 2015).
Just over half of the IRPs (20) involved people outside schools. The largest group of these groups comprised university science staff or students, acting as advisers/mentors. Examples in the USA include O'Neill and Polman (2004)    and one IRP programme taking place across six European countries (Dijkstra & Goedhart, 2011). Around a quarter of the IRPs included industrial partners and employers, e.g. Welch (2010) and Duran, Höft, Lawson, Medjahed, and Orady (2014). Less frequently, local voluntary groups and parents were involved, e.g. Adams et al. (2009). A small number of publications reported on IRPs undertaken by groups of schools or individual teachers in their own school and not involving any partners: Chin andChia (2004), Zion et al. (2004), Chien and Karlich (2007), Haigh (2007), Faris (2008), and Balmer (2014).
IRPs were most prevalent at upper high school level, i.e. for ages 16-19 (17 studies), as shown in Table 3.
Two models predominated for the creation of time for IRPs. Most commonly, they were undertaken during normal school hours, sometimes supplemented with time in after-school clubs (e.g. Brand, Collver, & Kasarda, 2008;Hong, Chen, & Hwang, 2013;Sahin, 2013). Typically, such IRPs were of six weeks to a year's duration (e.g. Chin & Chia, 2004;Dijkstra and Goedhart, 2011;Faris, 2008;Hong et al., 2013;O'Neill & Polman, 2004). Occasionally, time was created within schools through 'intensive pullouts' whereby students were taken off their normal timetable for a period to be dedicated to IRP work (e.g. Rivera Maulucci et al., 2014). Five of the IRPs were associated with dedicated out-of-school events such as one-or two-week summer schools and camps (e.g. Akinoglu, 2008;Burgin et al., 2012;Gibson & Chase, 2002;Metin & Leblebicioglu, 2011). In two cases (Brand et al., 2008;Yasar & Baker, 2003), the IRPs were linked to participation in science competitions or fairs.
There was only one country, Ireland, where the IRP work was a compulsory component of a national end-of-course science examination (Kennedy, 2014). In England, the IRPs could, for students aged 16 or over, optionally be entered for a national qualification (Daly & Pinot de Moira, 2010).
In just under one third of the cases (12), participation in the IRP was compulsory. In around half the IRPs (20), students participated as part of a team, with around a quarter (10) requiring individual participation. In a small number of instances, students could choose between team or individual participation. The majority of the IRPs required the generation of one or more products, as shown in Table 4. Written reports (19) and presentations (17) predominated, with many any IRPs requiring both. Occasionally, students were asked to produce a physical artefact or write a reflective diary.
Fifteen of the IRP programmes were supported by external funding. Characteristically, this was associated with work involving partnerships with universities, employers or other groups. Where work was required for external examination or specific to one school, it was unfunded. Typically, funding for IRPs came from grants secured from national funding organisations such as government, research councils and charitable bodies with an interest in science education, or from industrial sponsors. Examples included funding from BHP Billiton Most of the funding supported regional or local initiatives, though there were examples of national initiatives including, in Australia, the BHP Billiton funding, and, in the UK, the CREST awards 1 , the Nuffield Partnerships scheme 2 and the Royal Society's Partnership Grants scheme 3 .
Some studies reported data on traditionally under-represented groups in science, focusing on gender, ethnicity and socio-economic status, for example, in the USA, Yasar and Baker (2003), Sikes and Schwartz-Bloom (2009) and Sonnert, Michaels, and Sadler (2013), and in the UK, the Nuffield Foundation (2013) and the British Science Association (2014). In two cases in the USA (Duran et al., 2014;Rivera Maulucci et al., 2014), the IRP work formed part of a programme intentionally developed for students with backgrounds typically under-represented in science.

Findings: quality of evidence
In evaluating the quality of the evidence, some notes of caution need to be sounded in relation to factors that could result in bias in the evidence base. The review revealed that research into impact of IRPs was most often conducted by those associated with the funding agencies and with the development and/or running of the IRP, risking confirmation bias in the findings. Very few of the studies in this review had commissioned external evaluations, with Grant (2007) and Jenkins and Jeavans (2015) being exceptions. Another source of potential bias concerns the nature of the data collected. Frequent use is made of data provided by the people involved in IRP work. Many of the adults who can provide data (teachers, employers, and university-based scientists) are already likely to be very sympathetic to the aims of IRPs.
A challenge in synthesising the evidence arises from the diversity in provision and execution of IRPs, which is reflected in a corresponding diversity in the aims and range of measures used to assess impact.
The main sources of data were students and their teachers, with the most of the work focusing on the impacts on students. Impacts on understanding of concepts, practical skills, cross-disciplinary skills (e.g. working collaboratively in teams), attitudes towards science and motivation to continue with science after it was no longer compulsory were explored. There was also a cluster of studies focusing on impacts on traditionally under-represented groups in relation to gender, socio-economic status and ethnic background. Studies that focused on teachers and others involved (e.g. IRP providers, university scientists/mentors, employers, state/regional organisers) explored views of the impact of IRPs on students together with views on the potential benefits and drawbacks of their own participation in IRPs.  None of the studies employed randomised controlled trials; nine adopted some form of experimental design which involved making comparison between participants and nonparticipants in IRP activities (Finegold, 2015;Gibson & Chase, 2002;Jenkins & Jeavans, 2015;Krajcik & Blumenfeld, 2006;Moote et al., 2013;Sahin, 2013;Schneider et al., 2002;Welch, 2010;Yasar & Baker, 2003).
The predominant techniques used for gathering data from students were questionnaires, tests of understanding, inventories on affective aspects, and interviews and focus groups to explore students' views of IRPs. Occasionally, data were drawn from student presentations, student reports on their IRP work, student reflective diaries, observations of students undertaking IRP work, and datasets such as external test and examination results. Where quantitative data were gathered, it was very rare for the reports of studies to report details of any checks on reliability and validity with research instruments or data analysis. Table 6 summarises the impact outcome measures and the nature of the data gathered in the studies.
The wide variety of outcome measures points to one of the most prominent features of research into the impact of IRPs, which is the very disparate approach to judging the impact of IRPs.
Studies gathering data on cognitive impacts include those of Krajcik and Blumenfeld (2006), Burgin et al. (2012) and Sahin (2013) in the USA. Studies gathering data on the impact of IRPs on students' attitudes to science include Faris (2008) in Qatar, and Gibson and Chase (2002), Yasar and Baker (2003) and Welch (2010) in the USA. Other studies on affective responses include students' motivation (Moote et al., 2013), and students' self-efficacy (Sikes & Schwartz-Bloom, 2009). Other aspects explored include views of the nature of science (Metin & Leblebicioglu, 2011), development of students' practical and experimental skills (Chien & Karlich, 2007;Grant, 2007;Yasar & Baker, 2003;Zion et al., 2004), and development of more general skills in students, most often related to as collaborative working in teams (Charney et al., 2007;Faris, 2008;Grant, 2007).
Studies contained varying amounts of detail on the techniques employed to gather data on the impact of IRPs. As might be anticipated, full reports contained more detail than journal papers, particularly on instrument design. There were no examples of replication studies and virtually all the studies gathered data using specificallydesigned instruments, though a few used state or national test instruments of subject knowledge (Daly & Pinot de Moira, 2010;Krajcik & Blumenfeld, 2006;Schneider et al., 2002) or existing, validated instruments to measure student characteristics such as motivation (Moote et al., 2013). Most studies drew on at least two sources of data.
In order to evaluate the quality of the evidence base as a whole, the criteria widely employed in making such judgements about systematic reviews were used (see, for example, Gough et al., 2012). These take into account the declared aims of the studies, the hypotheses and research questions, strategies employed for identifying the sample, the nature and extent of the data gathered, the appropriateness of how the data were collected and the methods employed to analyse the data (including information on reliability and validity checks) and the extent to which the conclusions appear sound in relation to the data gathered.
With the exception of an over-reliance on self-reported data in some cases, the studies included in the review appeared to have adequate or good designs, with no obvious adverse effects arising from researcher involvement in the design and undertaking of studies. However, the quality of individual studies is offset by the diversity in focus and in the instruments used, with one of the most prominent features of the work as a whole being the comparatively uncoordinated and unsystematic approach to gathering evidence on, and judging the impact of, IRPs. This diversity poses a challenge for the synthesis of data, and the current evidence base would not permit a meta-analysis of data (i.e. an analysis that quantitatively aggregates data across studies, as is common in medical studies, leading to an overall conclusion as to the effects of IRPs with effect sizes or odds ratios).  (2013) USA * = more than one data source gathered in study.

Findings: the evidence on impact
A wide range of potential impacts of IRPs are reported, particularly on students, with studies reporting on students' responses to undertaking IRPs, effects on students' learning and effects on students' attitudes to science, including attitudes towards pursuing a career in science. Within this, some studies explore elements to do with widening participation issues. Some studies also report on impacts on teachers and, less frequently, impacts on other participating adults such as university scientists and employers.
All the studies exploring the impact of IRPs on students' interest in pursuing careers in science/science-related areas report increased numbers of students indicating that their participation in IRPs meant they were more likely to consider careers in science (e.g. Adams et al., 2009;Hubber et al., 2010;Jenkins & Jeavans, 2015). Students indicated that this was primarily due to increased awareness of the range of careers and the varied nature of work in which people with science qualifications engage. Improved practical skills and abilities are also reported (e.g. Adams, 2009 in the USA; British Science Association, 2014 in the UK).
One finding of particular interest to emerge from some studies in the USA and the UK was potential benefits to traditionally under-represented groups in science in relation to ethnicity and socio-economic status. In the USA, four studies report improved engagement for such students (Duran et al., 2014;Rivera Maulucci et al., 2014;Sonnert et al., 2013;Yasar & Baker, 2003), with Sikes and Schwartz-Bloom (2009) noting interest declining slightly. In the UK, the British Science Association (2014) found that uptake of IRPs was higher than average for students from lower socio-economic groups. The Nuffield Foundation (2013) reports particular benefits in engagement for students from disadvantaged backgrounds.
Where negative notes about involvement in IRPs were sounded in the studies, the focus was often on practical matters. These included teachers reporting that IRPs were unduly time-consuming (Faris, 2008), had a negative effect on the time available for teaching other aspects of science (Kennedy, 2014), or adversely affected the school's ability to deal with external inspections (British Science Association, 2014;Jenkins & Jeavans, 2015). Additionally, some teachers reported lacking the confidence to run IRPs (British Science Association, 2014;Jenkins & Jeavans, 2015) and problems with the identification of external partners (Jenkins & Jeavans, 2015). One study (Kennedy, 2014) reported that teachers felt IRPs discouraged students from considering further study of science subjects.
Very few details on assessment criteria for IRPs are reported in the studies, making it difficult to judge the evidence on assessment and hence measures of validity. This absence also hinders comparisons between the impact of more traditional approaches to practical work with that of IRPs, with none of the studies reporting on this aspect.

Conclusions
IRPs are often associated with country-wide policy initiatives in science education. Generally, they are perceived as valuable and important by a range of groups with interests or involvement in science education. Such groups include teachers, scientific researchers, industrial employers, charitable foundations, professional societies and learned bodies. The notion of allowing students to find out what it is like to be a scientist is seen as particularly attractive. In addition to the benefits reported for students, benefits are also claimed for other people involved, including the links built between students, teachers, schools and employers. However, the review shows that only a minority of students in any country are normally offered the opportunity to undertake an IRP.
The review reveals a diversity in the conceptualisation and implementation of IRPs, posing challenges for synthesising the evidence and making judgments about the impact and effectiveness. The review points to more work being needed to establish that the potential benefits of IRPs are such that they should definitely be used more widely in the school curriculum. A key element of such additional work would be focusing on improving the quality of the available evidence. This could be achieved by some relatively straightforward steps. First, the rigour of studies could be enhanced by making more use of experimental or quasi-experimental study designs. As IRPs are optional in most cases, this facilitates the gathering of data from control and intervention groups. Where such designs are not feasible, there are examples in this review (e.g. Krajcik & Blumenfeld, 2006) where good use has been made of existing data sets on student performance to enable comparisons to be made between students who have undertaken IRPs and wider populations. Secondly, there is an over-reliance in some studies on self-report data. More rigour would be introduced through the gathering of more than one source of data. Thirdly, and whilst recognising that methods of evaluation will need to vary in order to accommodate the particular features of specific IRPs types and programmes, those undertaking research on impact need to make greater use of previous work, particularly in relation to focus and methods employed to gather data. Considerable benefits would be conferred through greater agreement about the areas in which to collect data: increased use of existing, validated instruments, rather than excessive development of new instruments, would facilitate the building up of a more coherent evidence base. Areas of particular importance in which to gather data are students' learning of science concepts, students' views of the nature of science, and students' affective responses to participation in IRPs. In the second and third of these areas, a number of instruments already exist and could be utilised. The learning of science concepts is likely to require more work, given that many IRPs involve an indepth study of a relatively narrow area.
While this review has focused on the impact of IRPs on students, the studies have pointed to a number of other practical factors that would need to be considered were schools to be given more encouragement to offer IRPs. IRPs place particular demands on students, teachers, universities and employers that are not associated with more standard school provision. The demands relate to resource (time and money), skills required by teachers and other adults involved in IRPs, and supporting infrastructure. External funding to support IRPs currently comes from a range of sources, including government agencies, charitable bodies, industrial sponsors and other groups that fund research. There are some examples of co-ordinating bodies being established that could play a useful role in supporting IRP work (e.g. the Institute for Research In Schools [IRIS] in the UK), through identifying funding opportunities, training opportunities, and interested external partners.
In addition to the need for improvement in the quality of the evidence base, the review findings point to three particular areas that would benefit from further research to inform any decision on making more widespread use of IRPs. The first of these is the assessment of IRPs, where there is a dearth of information in the studies in the review. As noted earlier, one of the motives for undertaking IRPs is to give students the opportunity to 'be like a scientist'. Given this, it would be useful to involve practising scientists in discussions about what being like a scientist means in operational terms, and a consideration of what this means for assessment of IRPs. One avenue of potential utility is that of threshold concepts (Land, Meyer, & Smith, 2008;Meyer & Land, 2003), namely concepts that substantially change how students view their discipline and which change the learner's approach to, and perception of, learning in their subject. In the context of IRPs, threshold concepts are those concepts that are central to being able to think and act like a research scientist, which result in students seeing science in a new light, and may alter their feelings towards science. The second area of work relates to the emerging evidence of the possible benefits of IRPs for increasing engagement with science in students from traditionally under-represented groups. Here, there would be merit in undertaking case studies of particular groups of students in order to characterise the features of IRPs that appear to have a positive impact on views of science and engagement with science. Finally, given the range of short-term potential benefits reported for IRPs, it is important to explore possible longer-term benefits through looking at the impact of IRPs on subsequent subject and career choices. Notes 1. http://www.britishscienceassociation.org/crest-awards. 2. http://www.nuffieldfoundation.org/nuffield-research-placements. 3. https://royalsociety.org/grants-schemes-awards/grants/partnership-grants/.

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
This work was supported by the Wellcome Trust.