Challenges faced by manufacturers with clinical evaluation under the new European Medical Device Regulations

Abstract This study seeks to investigate the impact of strengthened requirements for clinical evaluation for medical device manufacturers in Europe due to the new Medical Device Regulations. Qualitative interviews were conducted with eight clinical evaluation consultants from eight different European countries who review and approve clinical evaluation reports for manufacturers prior to their submission to notified bodies. The study describes the sources of device clinical evaluation evidence and describes the consultants’ recommendations and challenges encountered when reviewing Clinical Evaluation Reports. The findings from the study demonstrate that understanding what constitutes sufficient clinical evidence poses the biggest challenge to the generation of an MDR-compliant Clinical Evaluation Report. Additionally, the study identified a knowledge and skills gap in the generation and assessment of acceptable regulatory and clinical data. Further, there is heterogeneity in the reviews of Clinical Evaluation Reports received by consultants and inadequate guidance to enable compliance by manufacturers. This study found that some manufacturers of certain CE-marked medical devices are planning to remove them from the EU market upon expiration of their certificate, and in the case of new innovative devices, some manufacturers may launch in other non-EU markets.


Introduction
Before placing a medical device on the EU market, the manufacturer must demonstrate to their notified body that the device is safe for use and performs as intended, or in other words, the medical device does not cause unexpected harm to a patient during normal use and functions as expected.A clinical evaluation report (CER) originated by the manufacturer's documents their devices supporting clinical evidence and conclusions regarding the safety and performance of the device and is a key document submitted to the notified body in support of CE marking (Medical Device Coordination Group, 2021a).Under European Medical Device Regulations clinical evaluation is defined as "a systematic and planned process to continuously generate, collect, analyse and assess the clinical data pertaining to a device in order to verify the safety and performance, including clinical benefits of the device when used as intended by the manufacturer" (EUR-Lex, 2021) Following an assessment of the Clinical Evaluation Report and other elements of the regulatory submission, if the notified body is satisfied that the manufacturer has demonstrated that the medical device is safe for use and performs as intended, the device may be placed on the market.Throughout the lifetime of the medical device, as more information and data are generated from real-world use of the device and collected via the manufacturer's Post-Market Surveillance (PMS) system, the manufacturer is expected to update the Clinical Evaluation Report and other relevant technical documentation.Real-world use generates real-world data (RWD) which is the data relating to patient health status or the delivery of health care routinely collected from a variety of sources (Federal Drugs Administration, 2023).Real-world data can come from a number of sources, for example electronic health records (EHRs), claims and billing activities, product and disease registries, patient-generated data including in home-use settings or data gathered from other sources that can inform on health status, such as mobile devices.Real-world evidence (RWE) is the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD.RWE can be generated by different study designs or analyses, including but not limited to, randomized trials, including large simple trials, pragmatic trials, and observational studies (prospective and/or retrospective) (Federal Drugs Administration, 2023).
Following a series of medical device scandals, the former Medical Device directives (MDD) were deemed not adequate for regulating digital and more complex/sophisticated medical devices; the directives were repealed and replaced by the new EU-MDR 2017/745 (MDR), which came fully into force on 26 May 2021 (Malvehy et al., 2022;Valla, 2021).
Since its entry into force on 26 May 2021, medical device manufacturers have initiated the process of transitioning to the MDR.Stricter clinical evaluation requirements introduced by the MDR may pose the biggest challenge for manufacturers, particularly as the amount and quality of clinical data that will be needed to demonstrate device safety and performance and to support the acceptability of the benefit: risk ratio is set to increase compared to the former European MDD (Holborow, 2021).Clinical data under the MDR is defined as "information concerning safety or performance that is generated from the use of a device" (EUR-Lex, 2021).Furthermore, as the notified bodies' assessment of the manufacturer's Clinical Evaluation Report is subject to review by the relevant competent authority and, in some cases, by expert panels, manufacturers may experience delays in their product reviews and greater demands on clinical data quality and quantity (Behan et al., 2017;Majety et al., 2021).
Historically manufacturers may have demonstrated equivalence to a marketed device in support of regulatory approval.However, under the MDR, it is now more difficult to rely on demonstrating equivalence to a marketed device, as the manufacturer has to generate their own clinical data in support of CE marking.
The new EU regulations on medical devices and in-vitro diagnostic devices allow devices CEmarked per the former EU directives to still be placed on the market under certain conditions.Such devices are referred to as "legacy devices" (Lacalle, 2023).Legacy Devices are specifically defined by the European commission as Medical Devices, Active Implantable Medical Devices and In Vitro Diagnostic Medical Devices that are covered by a valid certificate issued in accordance with Directive 93/42/EEC, Directive 90/385/EEC or Directive 98/79/EC and that continue to be placed on the market after the date of application of Regulation (EU) 2017/745 (MDR) or Regulation 2017/ 746 (IVDR) (European Commission :DG Sante, 2023).
For certain high-risk and legacy devices, as per article 65, clinical investigations may represent the only viable option for generating sufficient clinical data (Medical Device Coordination Group, 2021a).However, the high costs associated with clinical investigations can make them restrictive, particularly if the cost of generating the required clinical data for a particular medical device outweighs the potential return on investment.
The challenges faced by medical device manufacturers in meeting the strengthened clinical evaluation requirements pose a potential risk to the healthcare ecosystem as manufacturers may decide to discontinue certain medical devices leading to their removal from the EU market or to postpone the launch of innovative medical devices thereby preventing access to the patients that need them (Kearney & McDermott, 2023;Malvehy et al., 2022).Clinical evaluation is a complex and multifaceted topic.To the author's knowledge, there is no published data or study from a manufacturing consultant viewpoint identifying the specific elements of the clinical evaluation that pose the greatest barriers to ensuring clinical MDR compliance.Thus, this research project's value lies in providing a unique insight into the specific challenges faced by manufacturers addressing the strengthened clinical evaluation requirements from authoritative sources such as auditors and consultants.Based on the research outcome, the researcher aims to identify opportunities to develop targeted resource material to support manufacturers.
It is within this context that this research project aims to:   (Cohen, 2012) there were serious concerns about the stringency of European regulations.Investigations of these scandals raised issues around the approval processes for medical devices and notified body inspection and practices (Melvin & Torre, 2019) resulting in political pressures and the creation of a new more stringent MDR.

European medical device regulations changes -A summary
The MDR is a more lengthy and comprehensive document than its predecessor directives.For example, the MDD 93/42/EEC has 60 pages and the AIMD 90/385/EEC had 20 pages.To illustrate the changes required by manufacturers for conformance; Annex Chapter VI Article 61 on clinical evaluation within the MDR requires conformance to the general safety and performance requirements (GSPRs) while Annex XV Chapter VI Article 62 requires a clinical investigation to be conducted (European Commission, 2017).These added annexes strengthened Clinical evaluation requirements for high-risk and implantable devices and manufacturers must demonstrate evidence of the devices safety and performance via clinical investigations.The Clinical Evaluation Report was required to be updated for all types of devices by using Post Market Surveillance data and this data must be aligned to the risk management system (TUVSUD, 2021).
A clinical evaluation consultation procedure and inspection procedures are also introduced under the new MDR.These new procedures related to clinical evaluation consultation and inspection have been introduced in the MDR.These procedures are aiming to improve oversight of the notified bodies by the relevant county-specific competent authority by ensuring more inspection around the Notified Bodies evaluation of the manufacturer's Clinical Evaluation Report.For certain high-risk devices, the notified bodies' clinical evaluation assessment report must be reviewed by expert panels (European Commission, 2017).
Traditionally, Medtech manufacturers have preferred Europe as the market in which to launch their new devices followed by launching in the US market at a later stage (Speer, 2018;Vetalice, 2010).The US FDA has had more stringent demands for pre-market clinical data than Europe (Hwang et al., 2016;Vasiljeva et al., 2020).To place a medical device on the market in Europe and gain market authorisation the device's technical documentation, including the Clinical Evaluation Report, is submitted to the Notified Body to demonstrate device safety and performance (McGrane et al., 2022).If the Notified Body assesses the technical documentation and concludes that the device is safe and performs as intended in accordance with the manufacturer's intended use statement for the CE mark is placed on the device.Within the new MDR, there is increased inspection and oversight for Notified Bodies and a requirement for Notified Bodies to be designated to audit to the new EU MDR (NANDO, 2021).Thus, many Notified Bodies are no longer available to certify manufacturers and have not applied for MDR designation.This had led to an industry shortage of designated Notified Bodies for certifying to the MDR putting pressure on MedTech manufacturers in keeping their compliance.European MedTech manufacturers will also be required to increase reporting of Post-Market Surveillance performance utilising the European Medical Devices Database (EUDAMED) (European Commission :DG Sante, 2023).This extra reporting will also put resource constraints on the industry and increase compliance administration.

Sources used by manufacturers for clinical evaluation
Clinical evaluation is defined by the MDR as a systematic and planned process to continuously generate, collect, analyze and assess the clinical data pertaining to a device in order to verify the safety and performance, including clinical benefits, of the device when used as intended by the manufacturer (Eur-Lex, 2017;European Medicines Agency, 2018).The clinical evaluation process is highlighted in Figure 1.As depicted in Figure 2, clinical data is derived from multiple sources, including clinical investigations, peer-reviewed literature reviews, Post-Market Surveillance activities, including Post Market Clinical Follow-Up (PMCF) and vigilance activities (e.g.complaints, adverse events, serious incidents, undesirable side effects).In addition, data from the risk management process, particularly conclusions regarding the acceptability of the benefit-risk ratio, also feed into the clinical evaluation.
The strategy for identifying, appraising, and analyzing the clinical data must be carefully planned and documented in the Clinical Evaluation Plan (CEP), detailed in MDR Annex XIV Part A (European Medicines Agency, 2018).To draw valid conclusions, the clinical data's quality and scientific validity must be carefully assessed to determine its suitability.Therefore, the clinical evaluation must be completed by individuals or a multidisciplinary team with demonstratable competency (NANDO, 2021) The results and conclusions of the clinical evaluation and the supporting clinical evidence are documented in the Clinical Evaluation Report, and the notified body subsequently assesses it.This assessment aims to ensure that the manufacturer has provided sufficient evidence to demonstrate device safety and performance and to support their conclusions regarding the benefit: risk profile.As previously mentioned, the notified bodies' assessment of the manufacturer's Clinical Evaluation Report is subject to inspection by the relevant competent authority, and in turn, manufacturers can expect their Clinical Evaluation Report to be rigorously assessed by the notified body.Furthermore, for certain high-risk devices (Class III implantable and Class IIb devices intended to administer and/or remove medicine), the notified bodies' clinical evaluation assessment is also subject to additional inspection by an expert panel.In addition to post-CE marking, the Clinical Evaluation Report must be kept up to date using data collected through the Post-Market Surveillance system and is subject to further review as part of its notified body's surveillance activities (European Medicines Agency, 2018).
To ensure that the Clinical Evaluation Report can stand up to this level of inspection, manufacturers must implement a clinical evaluation strategy which is designed to obtain reliable, high quality and scientifically valid clinical data not only as part of initial CE marking but throughout the lifetime of the medical device; otherwise there is a risk of delays in the initial CE marking process or removal of the device from the market.

Clinical evaluation -pre-market clinical data sources
The primary clinical data sources for initial CE marking are peer-reviewed scientific literature and clinical investigations.Peer-reviewed scientific literature refers to articles written by experts in their field about a particular medical device or medical intervention.Literature reviews are a wellestablished route for generating clinical data.However, a limitation to this methodology is that peer-reviewed literature may not be available for the medical device itself, including the different variants, sizes and accessories, or for the specific clinical conditions that the device is intended to address, or there may be gaps in the data/information which prevent the manufacturer from drawing scientifically valid conclusions.The outcome of the literature review, including the protocol used, search engine, inclusion/exclusion criteria, article appraisal strategy, results, conclusions, etc., must be documented (Medical Device Coordination Group, 2021a).Any identified data gaps require supplementation with clinical data obtained from other sources

Clinical investigation
A clinical investigation refers to the formalized assessment of the safety and performance of a medical device in one or more human subjects.The results of high-quality clinical investigations (i.e.those conducted by ISO 14,155 Clinical investigation of medical devices for human subjects-Good clinical practice) represent the highest-ranked source of clinical data (Medical Device Coordination Group, 2021b).Under the directives, demonstration of device equivalency was commonly used to circumvent the need to conduct clinical investigations.Using this approach, manufacturers could claim that their device is at least as safe for use and has a similar level of performance as a currently CE-marked device or devices.However, under the MDR, demonstration of equivalence for implantable and class III medical devices requires the manufacturer to implement a contract with the second manufacturer granting them full access to the equivalent device's technical documentation on an ongoing basis and the equivalent devices clinical evaluation must be sufficient to demonstrate conformity with the MDR.As the second manufacturer may be a competitor, securing this contractual agreement will likely prove challenging.Therefore, a clinical investigation will be required for initial CE marking for implantable and class III medical devices.
In the case of Class IIa and IIb devices, equivalence to a CE-marked device or devices can be used without the need for a contract or agreement with the manufacturer of the marketed device.However, the Clinical Evaluation Report must contain sufficient clinical data to demonstrate that the device's technical, biological and chemical characteristics are similar to the marketed device and that there are no clinically significant differences in safety and performance.Accessing sufficient clinical data to demonstrate this level of equivalence may prove challenging, particularly in the absence of access to the marketed devices' technical documentation and the lack or absence of appropriate peer-reviewed published literature.Therefore, if the manufacturer cannot demonstrate equivalence, gaps in clinical data will need to be addressed via a clinical investigation (Medical Device Coordination Group, 2021a).
While clinical investigations provide the highest quality clinical data, they have limitations, as discussed above, particularly when evaluating long-term device safety and performance, and are not reflective of the actual environment under which the device will be used (FDA CDRH, 2016).In response to these challenges, the medical device sector are looking at alternative or supplementary approaches to traditional clinical investigations, including harnessing the power of data and information which is generated from real-world use of the medical device (Dang, 2023;Johnson et al., 2022).

Clinical evaluation -post-market clinical data sources
The primary sources of post-market clinical data include published scientific literature reviews and Post-Market Surveillance, including Post-Market Clinical Follow-up.Compared to the directives, MDR introduces detailed and more prescriptive requirements for Post-Market Surveillance and Post-Market Clinical Follow-up.To address these requirements, manufacturers will be tasked with updating and, perhaps, rethinking their current processes to ensure compliance with MDR.Further, for legacy devices, the requirements for Post-Market Surveillance and Post Market Clinical Follow-up are effective immediately from May 2021 and notified bodies will verify compliance as part of their MDD/AIMDD surveillance activities.
Peer-reviewed scientific articles can also describe the real-world clinical experience on the use of the device in support or otherwise of device safety, performance and clinical benefits and can also be used to identify the need for modification or changes to the original design of the medical device.
The Post-Market Surveillance system is an important element of the manufacturer's quality management system (QMS) and is designed to collect safety data, information, and experience from real-world use of the marketed device for the duration of its lifetime.An effective Post-Market Surveillance system allows the manufacturer to rapidly identify safety alerts associated with the medical device and implement necessary actions to ensure patient safety, including field safety corrective actions, product recalls, issuance of advisory notices, etc.The outputs of the Post-Market Surveillance system are also used to update the risk management file and clinical evaluation.Post-Market Surveillance activities can be reactive or proactive.Reactive Post-Market Surveillance activities can include analyzing customer complaints, serious incidents, field safety corrective actions, maintenance/service reports, performing literature reviews, reviewing Periodic Safety Update Reports (PSURs) and data trending.Proactive Post-Market Surveillance is when the manufacturer actively seeks data and information on their device using measures such as gathering feedback from focus groups, conducting patient/end-user surveys, monitoring recalls or other safety issues associated with similar medical devices and obtaining information from device registers and/or databases (Melvin & Torre, 2019).
Post Market Clinical Follow-up refers to the proactive collection and evaluation of clinical data from actual medical device use throughout its lifetime.It is a subset of the Post-Market Surveillance system and is used to bridge any gaps in knowledge regarding safety and performance and/or address new safety concerns and emerging risks identified as part of the Post-Market Surveillance activities or market surveillance activities.Clinical data generated through Post Market Clinical Follow-up is also used to (1) support the device benefit-risk ratio, (2) support the claimed lifetime (Holborow, 2021) and (3) verify that the intended purpose statement is correct (Medical Devices Coordination Group, 2020b).Post Market Clinical Follow-up is required for all devices unless a valid justification can be provided.PMCF data generated under the directives may not be sufficient to demonstrate compliance with the EU-MDR, particularly for devices that can no longer claim equivalence.Therefore, manufacturers of legacy devices need to consider the role of Post Market Clinical Follow-up activities as part of their EU-MDR transition strategy under the Medical Device Co-ordination group (MDCG) document MDCG 2020-6 (Medical Device Coordination Group, 2020b).An overview of the main types of Post Market Clinical Follow-up activities is presented in Figure 3. Surveys include questionnaires and interviews and are typically administered to the device user to gather data on usability, user experience, patient outcomes, opportunities to improve the device, etc. Surveys are less time-consuming and less expensive than Post Market Clinical Followup studies; however, as surveys can be subjective, the scientific validity of the resulting clinical evidence is typically regarded as lower quality than evidence derived from other sources (Medical Device Coordination Group, 2021b).

Methodology
A qualitative study was conducted using interviews with consultants who are clinical team leads from a mixture of continental European-based consultants from different consultancies.The qualitative interview model development consisted of four phases: 1.Preliminary phase,2.Questionnaire development, 3. Pilot testing and 4. Evaluation (Farnik & Aw Pierzchała, 2012).

Preliminary phase
The interviewees were identified and the research objectives were written.These individuals were identified in many cases from LinkedIn but some were known by reputation to the researchers from their own professional networks.There has been an increasing usage of social networking sites to identify individuals for research purposes (Antony et al., 2012;Tambe, 2014).It is found in some studies that current employment and career profiles are more accurate on the LinkedIn database than in other databases (Ge et al., 2016).LinkedIn is also the world's largest professional networking site with more users than any others.Potential participants were contacted prior to the interviews to request their participation.Only those consultants with hands-on experience of the clinical evaluation requirements under MDR were selected for interview.

Question development
Questions were developed based on the literature research of both peer reviewed journals and published Notified Body training material and Notified Body communications related to the MDR as well as including any relevant news media reports.Questions were developed in alignment with the research objectives of the study.Also it was important the questions were written in a clear manner and we easily understood.

Pilot study & conducting the interviews
Preliminary question testing or cognitive debriefing was used to ensure the understanding of the interview questions (Campanelli, 1997).The interviews were initially piloted with five industry professionals and consultants.Any comments related to the questions or length of the interview itself were taken on board.Once the pilot study was deemed positive the interviews commented.
When the interviewees were contacted the purpose of the study was fully explained and the intention to publish the study also reiterated.Complete anonymity was assured and the study research ethics information was shared with the potential participants.Ethical principles based on the respect of the researcher for all potential participants were followed and thus required the researcher to (i) obtain informed consent and (ii) maintain confidentiality (Patel et al., 2003).These interview discussions focused on identifying the most common discrepancies observed during the consultants clinical team review of the clinical evaluations submitted under the MDR.The interviews were ultimately held with eight clinical consultant team members responsible for assessing and consulting on the manufacturer's clinical evaluations.Notably, the researchers approached 18 Notified Body consultants but they declined to get involved due to concerns about "being seen to divulge client information" despite anonymity being assured and the study ethics being communicated (Hulley, 2007).Hence, this study is unique in that it is one of the first studies to get this level of clinical evaluation consultants participation (with anonymity assured) as these consultants effectively review manufacturers clinical evaluation material as if they were a notified body.Online interviews using Zoom or MS teams were conducted and took about 30-40 min.The interviewees freely gave of their time.Eight interviews were deemed enough as no new themes occurred after the sixth interview, and thus the sample was deemed saturated (Guest et al., 2006;Saunders et al., 2018).
The interview questions are outlined in Table 1.The interview questions were developed following the literature review related to the new MDR and the clinical evaluation requirements.Ten questions were asked and although some questions were specific, they were still open-ended so that the interviewees were encouraged to elaborate as much as possible (Elliott & Timulak, 2005).The interviewers made comments after each answer to summarize what was said and also to encourage more elaboration (Meyer & Willis, 2019).In particular, questions 9 and 10 allowed for more a more open-ended elaboration on their views as the questions asked referred to advice and help they could give manufacturers.

Evaluation of the interview findings
The interviews were transcribed once the recordings were available and identified and uploaded to Atlas Ti9 software for qualitative analysis using Interviewee numbers to maintain anonymity of the interviewees.The answers were classified under different themes (related to the questions) and open coding was carried out (Charmaz & Belgrave, 2015).Coding allowed the different thematic areas to be linked [47].Open coding was used to identify individual units, and axial coding was utilised to select major themes (Charmaz & Belgrave, 2015).Memoing was utilised to keep track of the different themes.Memoing serves to assist the researcher in making conceptual leaps from raw data to those abstractions that explain research phenomena in the context in which it is examined (Birks et al., 2008).
The use of memoing and the member-checking technique helped the researchers keep track of the themes whilst they were coding with three researchers enabling data verification ( , 2003).Inter-rater reliability was calculated to obtain consensus on the various themes or question related to the 10 sub questions for the interviews.The inter-rater reliability was found to be 94% which is deemed a satisfactory rating (Fleiss et al., 2003).Any differences were settled through discussion.
While a limitation of the study could be that there were only eight manufacturing consultant team members interviewed and those interviewed represented a wide range of consultants across Europe and thus were representative of the wider European Medtech manufacturing environment.While the consultants worked with al sized enterprises the majority of their clients were large enterprises as opposed to small and medium-sized enterprises.Also, the nature of qualitative research and the interpretation of such qualitative data by the researchers can be subjective (Chenail, 2011).However, the researchers have aimed to minimise bias by completing the following: detailed record keeping, ensuring interpretations of data are consistent and transparent; establishing and seeking out similarities and differences across interviewee accounts to ensure their different perspectives are represented (Noble & Smith, 2015;Sandelowski, 1993).Lastly, ensuring capture of rich verbatim descriptions of interviewee accounts supports the findings (Slevin & Sines, 1999).

Results
The first question asked of the interviewees was, "Based on your experience, what are the key challenges faced by manufacturers meeting clinical evaluation requirements?"There were 37 individual challenges identified (Table 2), and the %'s of each challenge are outlined in Figure 4. Thirty-two percent of the challenges were categorized as related to a skills/ knowledge gap, particularly in understanding the requirements of the MDR.For example, one consultant interviewee stated that the biggest challenge is "Understanding the requirements (MDR requirements and Medical Device Co-ordination group (MDCG) guidance documents)".This is very evident when you review their clinical evaluations-they do not know how to give information so that the Notified Body understands it and do not know what they are supposed to provide".
Sixteen percent of the challenges identified related to obtaining and understanding the level of clinical data required.One interviewee responded that, based on their experience, "Some manufacturers are reluctant to change their approach in relation to what is acceptable in terms of clinical data", and another interviewee responded, "that the feedback they receive from the auditee is that "our Notified Body auditor did not request this level of data before" and "therefore manufacturers are willing to take a chance".
Other challenges included lack of resources, quality of documentation, inconsistency in the documented information, lack of interaction by manufacturers with notified bodies prior to and during their submissions to clarify any potential queries, resulting in inconsistencies in the application process and poor communication with and from Notified Bodies.A summary of the themes is presented in Figure 4.

Is there any specific clinical evaluation pathway, e.g. Article 6 or other specific articles that cause problems?
MDR Article 61 describes several different clinical evaluation pathways depending on the type of device (NSAI, 2022).On review of the responses to this question, three consultant's report that manufacturers lack clarity on when to apply Article 61 subsection 10 on the handling of devices where clinical data is not deemed appropriate or devices which do not require clinical data.For example, the British Standards Institute (BSI) offers an example of a medical refrigerator for storing tissues as a device that would not require clinical data and could apply Article 61 subsection 10 (Holborow, 2022).Typically, a device qualifying under article 61 subsection 10 would not have interaction with the human body.

Solutions
Scientific Literature Review • Search period and start/end date are absent • Search terms are inadequate • Search limited to one database • No consideration of similar/ equivalent devices • Missing inclusion and exclusion criteria • No appraisal of the literature • Search is not reproducible • Only favourable data is included • Literature is outdated • Improved training and knowledge of how to carry out a literature review.
• Improved guidance Post Market Clinical Follow-up (PMCF) Activities • Post-Market Clinical Follow-Up (PMCF) study conducted using a different device • PMCF not conducted within the intended use of the device A further 3 consultants reported that application of Article 61 subsection 8 on wellestablished technologies is problematic.Article 61 (8) refers to where there are wellestablished technologies that a manufacturer may claim exemptions to clinical trials.For example according to the MDR, a new device never previously marketed under Directives can be considered a Well Established Technology if it is sutures, staples, dental fillings, dental braces, tooth crowns, screws, wedges, plates, wires, pins, clips and connectors and is Class III or Implantable (Holborow, 2022).
Four consultants had the experience of working with manufacturers who faced challenges with legacy devices; however, this was more related to the manufacturers lack understanding of what data was required rather than a particular article or pathway.• Inclusion and exclusion criteria not defined • Document control issues: Absent signatures, dates and version control • No information on clinically relevant changes made to the device • A clinical development plan is absent for legacy devices • No discussion on applicable General Safety and Performance Requirements (GSPRs) • For legacy devices, relying on clinical data is used for Conformité Européene (CE) marking under the directives.
• Adequate training and availability of resources • Adequate guidance from regulators

Have you noticed an increase in outsourcing of clinical evaluation activities since the EU-MDR was introduced compared to the EU-MDD, and if so, why are companies outsourcing?
All (100%) interviewees reported an increase in clinical evaluation outsourcing.However, reviewing the responses, this increase appears to be attributed to a lack of internal knowledge and expertise.Some responses in relation to this question included: "There is a shortage of resources with the required experience and background on the market", "Emphasis on clinical evaluation and Post Market Clinical Follow-up has led to an increase", and "Clinical evaluation takes much longer to complete now compared to the MDD and manufacturers are struggling to hire the people they need".

What mistakes do companies make with the clinical evaluation report?
More than 20 individual mistakes were identified, of which the vast majority (nearly 60% of responses referred to this issue) related to the provision of adequate supporting clinical data, with one interviewee stating that often the "clinical research does not support the claims they are making".Another interviewee highlighted that a "process for collating, synthesizing, and analyzing data to generate accurate and reliable conclusions is needed-companies do not know how to establish this process".This is an important consideration, as the generation of a regulatory acceptable clinical evaluation report relies on effective interaction and flow of information and data between the clinical evaluation and other QMS processes (e.g.PMS including complaints and vigilance, Post Market Clinical Follow-up, and risk management).
Nearly one-third of the mistakes highlighted are related to the generation of poorly structured Clinical Evaluation Reports which are challenging for external reviewers to follow.According to interviewees from the consultant group, this directly leads to an increase in the number of queries raised during the notified body review, thereby extending the product review time.The third most frequent mistake (14%) relates to the use of the clinical evaluation developed under the MDD for submission under the MDR.One interviewee indicated that based on their experience, "this was more frequent for larger companies to try to retrofit their old clinical evaluation to address MDR requirements".

What are the common proactive and reactive post-market surveillance data sources?
The interviewees report that the use of Complaints data, Corrective Actions/Preventive Actions, Vigilance data and Recalls represent the most widely used sources of reactive Post-Market Surveillance data.In contrast, regulatory authority databases and Post Market Clinical Follow-up activities (Post Market Clinical Follow-up patient and clinician surveys and Post Market Clinical Follow-up studies) represent the most widely used proactive sources.However, one interviewee noted the impact of COVID-19 "Very difficult to get into hospitals to do Post Market Clinical Followup due to COVID-19."

Have you experienced companies removing devices from the market due to EU-MDR clinical evaluation requirements?
Most of the interviewed consultants had the experience of manufacturers deciding to remove devices from the EU market due to difficulties meeting the strengthened clinical evaluation requirements.One consultant interviewee stated that "Manufacturers will need to consolidate portfolios and discontinue devices in Europe which may lead to acquisitions/mergers to claim equivalence".Another interviewee provided additional insight into the potential strategy being employed by larger manufacturers, "There is talk at conferences from bigger manufacturers that they intend to go to the US first for approval for new devices and then go to the EU.However, this decision is not only based on EU-MDR; they are also looking at the market potential for the device (the US may be a bigger market potential).So it is a combination of market potential and regulatory requirements".One consultant stated that based on their experience, "No devices removed yet; however, restrictions applied to indication for use/clinical: benefits changed/claims changed on the website".This highlights the importance of ensuring that sufficient clinical evidence is available to support all claims documented in the device's intended use statement and other marketing/ promotional material.

What advice would you give a manufacturer today?
The interviewees provided 11 recommendations for manufacturers preparing for CE marking under MDR.These 11 recommendations fall into 3 main categories.Among the recommendations, developing an appropriate clinical evaluation strategy which includes reviewing the current product portfolio, identifying and ranking gaps in clinical data and development of appropriate action plan to address those gaps, represents the top recommendation, with one consultant recommending that manufacturers start by identifying "Which of my products do I realistically believe I can get sufficient data for (quality and quantity)-focus on this 10-20% of product portfolio?" Other recommendations included preparing high-quality product submissions and "answer queries from the notified body clearly and in detail so that you do not waste time" and lastly, aligned with the challenges presented in Table 1 on challenges, investment in the recruitment of suitably competent resources (both internal and external) was also listed as a key recommendation.

What can a notified body do to help medical device manufacturers(in your opinion as a consultant)?
The purpose of this question was to enable the researcher to identify opportunities to develop guidance documents and other resource materials to support medical device manufacturers with their product submissions under MDR.A summary of the responses is presented in Table 3.
On review of the list, recommendations such as pre-submission meetings/audits are not permitted under EU-MDR Annex VII.In addition, commentary on the lack of consistency between European and international regulatory agencies is outside this research project's scope.However, it should be noted that within the EU, the relevant competent authority is responsible for maintaining oversight of the notified bodies located in each EU member state.In addition, once EUDAMED goes live, decisionmaking by each notified body and data on device safety and performance will be publicly available.

Challenges to creating clinical evidence
The quantity of peer-reviewed scientific literature specifically addressing the research topic was minimal.However, several authors suggest that meeting the requirement to generate sufficient clinical evidence in support of CE marking may pose the greatest challenge for manufacturers (Holborow, 2021;Majety et al., 2021).The literature review also determined that in the absence of a cost-effective approach for generating the required clinical evidence, some manufacturers are expected to decide not to pursue MDR CE marking for certain devices (Malvehy et al., 2022).
The data generated from this research project corroborate these findings, whereby irrespective of the risk classification of the device, the determination of what constitutes sufficient clinical evidence represents the main challenge when generating an MDR-compliant Clinical Evaluation Report.Furthermore, the results from the consultant interviews further highlight the extent of this challenge, with 57% of mistakes during the preparation of the Clinical Evaluation Report related to the use of poor quality and inadequate supporting clinical data.
In the absence of a specific MDR definition for what constitutes sufficient clinical evidence, MDCG 2020-6 aims to guide by clarifying that "sufficient clinical evidence" is the "present result of the qualified assessment which has concluded that the device is safe and achieves the intended benefits" (Medical Device Coordination Group, 2020b).Qualified assessment implies that certain competencies are required to generate an MDR-compliant clinical evaluation and identify potential data gaps that require remediation.Identifying personnel with the required competency was identified during consultant interviews as one of the main challenges faced by manufacturers.The impact of not understanding the MDR clinical evaluation requirements is perhaps exemplified by the main discrepancies identified during the consultants assessment of the manufacturer's clinical evaluation, including "relying on clinical data used for CE marking under the directives" and "Clinical evaluation is not updated with new clinical data obtained via Post-Market Surveillance and Post Market Clinical Follow-up activities".The findings from the consultant interviews also demonstrate that some manufacturers of legacy devices are erroneously using clinical data generated as part of initial CE marking under the directives to generate their MDR Clinical Evaluation Report.Therefore, to ensure timely approval of the device's Clinical Evaluation Report by the notified body, manufacturers must ensure that personnel receive appropriate training on the MDR clinical evaluation requirements and that they appreciate the very real possibility that the clinical data generated under the directives are unlikely to address the MDR requirements (Medical Device Coordination Group, 2021b).
To aid in their understanding of the clinical evaluation requirements, manufacturers of all devices, irrespective of their risk classification, can refer to publicly available Medical Device Co-ordination group (MDCG) guidance documents, particularly MDCG 2020-13 and MDCG 2020-6.MDCG 2020-13 provides a template for use by notified bodies during their assessment of the manufacturer's clinical evaluation (Medical Device Coordination Group, 2020a), whereas MDCG 2020-6 provides guidance on what constitutes sufficient clinical evidence for legacy devices.The MDCG also published MDCG 2022-11 a position paper titled "Notice to manufacturers to ensure timely compliance with MDR requirements" in June 2022 advising engagement with notified bodies by manufacturers on their MDR requirements and compliance (Medical Device Coordination Group, 2022).However, despite these guidances's, in the absence of clarification from the European Commission regarding their expectations on what constitutes sufficient clinical evidence for specific types/groups of devices, confusion regarding how much clinical data is required is likely to continue resulting from inconsistent demands being placed on the notified bodies from each individual competent authority, and these, in turn, being passed down to the manufacturer.The device's clinical evaluation and the supporting documents (risk file, Summary Statement of Clinical Performance, Post Market Clinical Follow-up, Periodic Safety Update Report, Instructions for Use, etc.) that feed into the clinical evaluation must be continuously updated throughout the lifetime of the medical device based on data and information gathered during post-market use.Lack of consistency across these documents, most notably in the intended use statement, was identified as a common deficiency by the consultants.The intended use statement is a key element of the device's clinical evaluation and forms the basis for identifying the clinical evidence required to support the manufacturer's claims.In the absence of a clearly defined and consistently documented intended use statement for which sufficient supporting clinical evidence is available, the CE marking process will be delayed pending the resolution of the notified body-issued queries.
Given that the length of time taken for the notified body to complete their assessment of the clinical evaluation report is already expected to increase under the MDR (Majety et al., 2021) manufacturers must ensure that their supporting documentation complies with the expectations of the MDR and is fit for regulatory submission.
The consultant interviews point towards an absence of "internal communication and lack of different functional interactions" in relation to preparedness.The MDR enhances the requirements for demonstrating device safety and performance throughout the lifetime of the medical device.For manufacturers, this may drive the need to conduct an internal review of their operations and, if required, adapt their departmental structure to ensure a more collaborative process-based approach.Adoption of a process-based approach, which is also promoted by ISO 13,485:2016 and other quality management system standards, would (1) facilitate manufacturers with the implementation of a total product life cycle approach, (2) enable more efficient communication leading to earlier detection of potential safety signals requiring field safety corrective actions and/ or needs to improve/amend the device design, and (3) ultimately improve consistency and traceability throughout the device's technical documentation.
To address some of the challenges they are facing, manufacturers may outsource the generation of their device's clinical evaluation to an external consultant.Given their role, it is not surprising that all the consultants interviewed for this study reported an increase in outsourcing since the introduction of the MDR, with several reporting the need to recruit additional resources to meet the demand.For manufacturers, outsourcing offers several advantages, including upskilling internal staff through knowledge transfer and more timely and effective generation of regulatory documentation, which are "right first time" and less prone to delays due to notified body queries.
The need to outsource the generation of their Clinical Evaluation Report to external clinical evaluation consultants due to a lack of internal expertise may place a restrictive financial burden on these organizations, thereby limiting the availability of innovative medical devices for improving patient quality of life.Hence, the EU expects to face a shortage in the supply of certain medical devices (Medical Device Coordination Group, 2021b).
In relation to new devices, despite the US representing the largest global medical device market, in the past, many manufacturers made the strategic decision to launch new devices in the EU ahead of the US.This decision has been attributed to several reasons, including lower clinical data requirements, faster approval time and lower costs (Center for Drug Evaluation & Research, 2020).However, introducing stricter MDR requirements for clinical evaluation will force many manufacturers, according to the consultants, particularly high-risk devices, to rethink their global strategy.The potential impact of this strategy is a delay in access to new and innovative medical devices for EU patients (Melvin, 2022).It is important to note that the public and regulators will not know if devices are innovative or dangerous without clinical data, hence its importance irrespective of it effect on the manufacturers global product launch strategy.
Thus, the main challenges faced by manufacturers when generating an MDR-compliant Clinical Evaluation Report are determining the quantity and quality of clinical data required to generate sufficient clinical evidence and understanding the expectations of the MDR.Due to these challenges, the EU market is becoming less attractive for medical device manufacturers.
The generation of an MDR-compliant Clinical Evaluation Report relies on collecting, appraising, and assessing a sufficient quantity of high-quality clinical data.For legacy devices, the strength of the clinical data varies considerably, meaning that some clinical data sources are more suitable for demonstrating MDR compliance than others (Medical Device Coordination Group, 2021b).The findings from this study indicate that irrespective of device classification, supporting clinical data for legacy devices is primarily sourced from the manufacturer's Post-Market Surveillance system, including Post Market Clinical Follow-up activities, scientific literature reviews and collecting RWD.The need to conduct a new clinical investigation is driven by a significant gap in clinical data and puts additional time and financial pressure on the manufacturer.
In agreement with the findings from the consultant interviewees, complaints and analysis of vigilance data (serious incidents, Field Safety Corrective Actions, Field Safety Notices) represent the most widely used sources of reactive Post-Market Surveillance data.However, whilst these data sources are useful for the early detection of safety and performance signals, their use as a source of clinical evidence is limited, as evidenced by the low ranking assigned to the quality of the collected data (Medical Device Coordination Group, 2021b).Scientific literature reviews were identified as the third most commonly used source of reactive Post-Market Clinical Follow-up data.According to Medical Device Co-ordination group -MDCG 2020-6, for legacy devices, data derived from this source is generally regarded as an acceptable source of clinical evidence, provided the resulting data is of sufficient quantity and quality (Medical Device Coordination Group, 2020b).
MEDDEV 2.7/1 Rev 4 guides the conduct of a scientific literature review.Despite the widespread use of MEDDEV 2.71/1 Rev 4, consultants reported several challenges they saw to the conduct of a regulatory acceptable scientific literature review.One of the limitations of this methodology is that relevant scientific literature may not be available, thereby presenting a challenge for researchers regarding ensuring coverage of similar/equivalent devices.To increase the chances of identifying all relevant articles and data, MEDDEV 2.71/1 Rev 4 recommends using several databases, namely PubMed, EMBASE, Cochrane, and internet searches which could include the use of Google Scholar.However, conducting multi-database searchers is time-consuming and laborious (Bramer et al., 2017), perhaps explaining why the "use of one database" was reported as a common deficiency by the consultants in assessment of the manufacturer's literature review.
Also published clinical trials in peer-reviewed journals are not always 100% dependable in what they report and how their findings are interpreted.There may be violations of procedures whether unintended or intended, or misreporting and perhaps unfounded conclusions (Bustin & Nolan, 2016).Hence, the FDA requires raw data for review rather than published papers and performs its own analysis and interpretation on what these data say (Centre for Devices & Radiological Health, 2020).
In summary, Clinical data can vary in terms of what is offered and how it can be interpreted and thus how useful it can be in EU-MDR 2017/745 applications.Manufacturers may provide direct data and information from a strictly conducted clinical trial following an ISO standard e.g., ISO 14,155 or indirect data if they present data from a study designed by an investigator according to their objectives and design and thus may be supportive if appropriately interpreted or not.The aforementioned data sources reinforce the importance of verification by Notified Bodies.

Sources of clinical data
The development of an appraisal strategy was identified as a challenge to conducting a literature review and a deficiency noted by the consultants.Interestingly, guidance on the development of a literature appraisal strategy is addressed by the International Medical Device Regulators Forum clinical evaluation document (IMDRF, 2023).Given the importance of literature reviews as a source of clinical data, these findings highlight the need for the Medical Device Co-ordination group (and/ or notified bodies to develop an appropriate guidance document on conducting a regulatory acceptable literature search protocol.

Periodic safety update report review
Annex III of the MDR requires manufacturers to collect Post-Market Surveillance data by reviewing the device's Periodic Safety Update Reports.Given that the Post-Market Surveillance requirements, including the generation of a Periodic Safety Update Report, have been applicable since May 2021 and notified bodies are responsible for verifying that the manufacturer has implemented the required processes during surveillance audits (MDCG 2021-25), it was surprising that a review of Periodic Safety Update Reports was also not identified by any of the interviewed consultants.One possible explanation could be that at the time of data collection, whilst all respondents may have established the process for generation of the Periodic Safety Update Reports, only those manufacturers who have obtained CE marking under the MDR prior to May 2021 are actively conducting the annual (class IIb and III) or biennial (class IIa) review and update of their Periodic Safety Update Reports.For other manufacturers, the focus may be gathering the required clinical evidence and preparing their regulatory submissions to support CE marking.

Proactive post-market surveillance data
In agreement with the findings from the consultant interviewees, this study found that monitoring product databases are the most widely used approach for gathering proactive Post-Market Surveillance data, followed by Post Market Clinical Follow-up activities (both surveys and studies).

Post market clinical follow-up activities
Post Market Clinical Follow-up activities fall into two categories, namely surveys and studies.According to MDCG 2020-6, compared to a PMCF survey, the clinical evidence generated from a Post Market Clinical Follow-up study, including RWD collected from device registries, is of higher quality and is, thus, more useful for demonstrating MDR compliance (Medical Device Coordination Group, 2020b).However, during the consultants assessment of the manufacturer's clinical evaluation, deficiencies in the submitted Post Market Clinical Follow-up study plan and Post-Market Clinical Follow-up evaluation report were noted, including failure to update the other QMS processes, namely complaint handling, vigilance, risk management, technical documentation, with information derived from the Post Market Clinical Follow-up study.These findings further support the importance of operating a process-based QMS with appropriately defined communication and feedback channels and ensuring that personnel understand the expectations of the MDR regarding a continuous update of the appropriate technical documentation files.

Conclusion & future research
Through the analysis and discussion of the research findings, the researcher answered each of the proposed research questions in relation to 1. investigating the challenges faced by medical device manufacturers when generating their clinical evaluation under EU-MDR 2017/745 and 2. Identifying the main data sources employed by medical device manufacturers to collect information on the use of their device and thus achieved the study aims detailed in section.A limitation of the study was the small sample size within the study but the results did demonstrate data saturation in terms of opinions with no new themes emerging thus validating the opinions and results expressed.
Therefore, the study's conclusions relative to each of the project aims are as follows: (1) Irrespective of device risk classification, determining the level of clinical evidence required to demonstrate compliance with the MDR poses the greatest challenge for medical device manufacturers.This is further exacerbated by the lack of consistency regarding the level of evidence that different notified bodies are willing to accept.While the enhanced requirements are intended to ensure that only safe and effective medical devices are placed on the EU market, challenges addressing the requirements will lead to removing some medical devices from the EU market.In certain cases, there is a delay in patient access to innovative devices as manufacturers look to other markets to launch new devices before launching in the EU.
(2) Generation of an MDR-compliant Clinical Evaluation Report is a complex process involving collecting, appraising and assessing clinical data derived from multiple QMS processes.Personnel involved in this process must have an in-depth understanding of the MDR requirements governing clinical evaluation and the linkages and feedback loops between clinical evaluation and the other QMS processes such as Post-Market Surveillance, risk management, vigilance, data analysis and trending, etc.This study's findings indicate that manufacturers use several different approaches, including referring to Medical Device Co-ordination group (guidance documents, internal knowledge transfer, notified body publications and/or external training) to upskill their employees in this area.However, a potential knowledge gap exists when it comes to understanding the interface between regulatory affairs and QMS process interaction, understanding the strength and limitations of different clinical data sources, and conducting a regulatory-acceptable literature review.Identifying these gaps provides notified bodies and the Medical Device Co-ordination group (with opportunities to develop guidance in these areas and educational training providers to target their course offerings to address these gaps).
It should be noted that on the 7 th of March 2023, the European Council (post completion of this research study) voted 27-0 to adopt the measure put forward by the European Parliament and Commission to push back the MDR transition period.The EU cited the reasons for the push back as due to concerns about the supply of medical devices, notified body capacity and manufacturer's preparedness which could potentially avoiding forcing tens of thousands of products off the market in the European economic commission region (Al-Faruque, 2023a).The concerns which the EU has accepted as valid correlate with concerns raised in this study.
The implications of this study are many from a theoretical and managerial viewpoint.This study will be of value to both manufacturers in understanding the Notified Body requirements and the requirements of Clinical Evaluation Report under the MDR and European Regulators in understanding the challenges that manufacturers are currently stating.Furthermore, this is the only study of its kind to the author's knowledge of the challenges of clinical evaluation under the MDR and from experts such as consultants who have a high level of access to manufacturers comparable to that of notified bodies.This study highlights the implications for both manufacturers and patients within the EU market.The purpose of medical device manufacturing is to improve health and healthcare.The study demonstrates concerns in relation to how manufacturers are intentionally or unintentionally unaware or ignoring regulations and proper clinical research methodologies.The study demonstrates that while regulators need to improve the clarity of regulations, associated guidelines, expectations, and relevant documentation and be available as a resource for clarification, the onus for determining the safety and performance of devices is still ultimately with the manufacturers.
Future research topics will be in relation to the following: • What is the impact of device removals from the MDR enforcement on patient quality of life in the EU and other jurisdictions and global markets that use CE marking?
• Given the small size of the study population and the potential bias towards large manufacturers, it would be interesting to explore the topic in a larger sample size to determine if the identified challenges are similar or, indeed, more pronounced for SME organizations.
• It is of interest to further explore the reason for the reported inconsistencies in the amount of clinical data accepted by different notified bodies and the potential impact on the ability of the EU to maintain its position as an attractive market for medical device manufacturers.
• A further study on whether those devices removed from the EU market prior to MDR certification are for a genuine lack of data whether it be a lack of data to verify their safety/performance or due to purely product lifecycle and economic reasons.
(a) Investigate the challenges faced by medical device manufacturers when generating their clinical evaluation under EU-MDR 2017/745 (b) What are the main data sources medical device manufacturers employ to collect information on the use of their devices?

Figure
Figure 1.Clinical evaluation process.The clinical evaluation process can be described in 6 steps, from establishing a clinical evaluation plan through to preparing the clinical evaluation report.The process is repeated throughout the lifetime of the medical device.The clinical evaluation report must be kept up to date with sufficient clinical evidence in support of device safety and performance and confirmation of the acceptability of the benefit-risk profile.

Figure
Figure 2. Clinical data sources.Clinical data can be derived from various sources, including clinical investigations, peerreviewed scientific literature, Post Market Surveillance (PMS) activities, Post Market clinical follow-up (PMCF) and vigilance activities, and data from the device's risk management file.
Figure 3. Post Market clinical follow up (PMCF) activities are typically categorized as a study or survey.PMCF studies are designed to answer specific safety and performance questions and are conducted using a planned and systematic approach.Surveys are typically administered via questionnaires to patients and/or medical practitioners or other relevant device users to gather data and information on user experience.

Figure
Figure 4. Common challenges made by medical devices manufacturers during the preparation of the clinical evaluation report.

Table 1 . Qualitative interview questions Number Question Sources in Literature
Creswell &

Table 3 . Summary of recommendations provided by each interviewee on how a notified body can assist medical device manufacturers, Interviewee Recommendations
Perhaps harmonize an approach similar to the FDA with Complete Response Letters (CRLs) etc.For example, set up Meetings A, B, and C types before a review that is setup a Type A meeting at least 2 weeks before the formal meeting, a Type B meeting, at least 4 weeks before the formal meeting or a Type C meeting at least 4 weeks before the formal meeting Perhaps provide a fixed table of contents for the Annex II subset of International Medical Device Regulators Forum (IMDRF) International Notified Bodies are more lenient compared to pure European Notified bodies.Tell companies what you expect to see (not how you would do something) -a company could pay for a meeting (companies submit questions in advance).Responses to submissions (3 rounds) -standardize responses so that the client knows exactly what the gap is and what is needed.Clients are not clear on what documents are required.The notified body could create a list of a generic set of documents that the client needs to submit.