European Union Digital education framework: A quality standard to guide the design of healthcare apps

Abstract Digital Education Initiatives and Timely Solutions (DIG-IT) is an Erasmus+ project collaboration between university academics, clinical educators, industry partners, healthcare professionals, and technology experts over 5 European Union (EU) states. The objectives are to create digital educational capacity for academic faculty and mobile learning opportunities for continuing professional healthcare education. The first deliverable of this project was the innovative European Union Digital Education Quality Standards Framework and Toolkit or EU-DEF. This framework can be used to consider all principle and secondary aspects when designing, delivering, and evaluating digital education resources. In this article we explore the framework efficacy in collaboratively developing two open access, mobile healthcare apps, ECG Interpretation for Nurses and Nurse Leadership and Management. We describe the teamwork needed and steps taken during this process, and demonstrate the adaptability, usability, and value of this tool for other online development projects. The EU-DEF is an evidence-based, comprehensive framework that can be used as a starting point for individuals or teams when developing digital educational projects. Considering all variables helps focus the expertise required to produce robust end-products.

to provide accessible, convenient, teaching and learning at the higher education level and for continuing professional development (CPD).
To date the project outputs include a digital education quality standard framework and companion assessment toolkit (MacDonald et al., 2021a); a nine-module online course to increase confidence and teach educators to design, deliver, and evaluate digital education (MacDonald et al., 2021b); a bilingual (English & Italian) train-the-trainer online course for educators who have graduated from the nine-module course and want to be digital education champions for their department or faculty, an open access repository to share resources, and two bilingual (English and Italian) healthcare online mobile applications. Many of these activities occurred simultaneously from 2019-2021 and two papers sharing these resources and findings have been published. A third deliverable was the development of two healthcare educational apps. This paper describes the collaborative processes used by the DIG-IT team to create and provide open access to two online, mobile healthcare applications designed using the EU Digital Education Framework as a quality standard. The framework provides a comprehensive diagrammatic, visual, and explanatory foundation that considers essential requirements for developing online learning opportunities, particularly programs and courses where there is student/student, student/instructor interaction.

Framework structure
Digital education frameworks identify, explain, predict, and demonstrate complex relationships between concepts, key and sub-variables, and best practices of digital educational phenomena (MacDonald et al., 2021a). Their development is guided by philosophical underpinnings and empirical research. To help standardise digital education in line with technological advances, an appropriate framework was required. The DIG-IT project team assessed existing digital, e-learning, and online frameworks to ascertain if any were comprehensive enough to suit existing digital requirements. Initial research included an integrative literature review that explored several existing frameworks to determine their applicability to effective digital education in the EU in 2020 and beyond (MacDonald et al., 2021a).
Results of this study indicated that although educational frameworks were plentiful, few considered all components, dimensions, and perspectives of digital education. Thus, the DIG-IT team extracted relevant variables and sub-variables from existing frameworks, and with input from academic and technological experts, created the flexible, futuristic, EU-DEF that considered all aspects of developing a full range of digital education products such as program courses, and educational apps.
The EU-DEF acknowledges that one's learning is an interactive, subjective, and ongoing process. Individuals' cognitive processes differ and their perceptions of, and reflections on their environment and experiences are unique and linked to prior knowledge. As such, learning is fluid and contextual. Mental states and thought processes are always evolving as learners make sense of their world, individually and collectively (Bates, 2019). This is certainly the case in continuing healthcare professional education where technology reduces the half-life of medical knowledge (University for Digital Technologies in Medicine and Dentistry, 2020) yet competencies must be current. This reality underpinned the conception for the DIG-IT healthcare app initiative. During development, the EU-DEF was used as a credible, quality standard and guide for designing, delivering, and evaluating effective digital education experiences.
The EU-DEF proposes five critical dimensions required to elicit desired outcomes of an online product or event: structure, community, content, delivery, and support, of which organizational change is the pinnacle. Variables and sub-variables in the framework were identified from an extensive review of literature of existing digital education frameworks and models. Definitions of the variables and sub-variables can be easily accessed by opening the framework and clicking on the desired explanation. Using this tool is uncomplicated no matter what educational resource is being created. The following explanation for development of the apps starts at the bottom of the framework, Structure and works through the levels: Community, Delivery, Outcome (Figure 1).
From MacDonald et al. (2021b).uropean Union Digital Education quality standard framework and companion evaluation toolkit, Open Learning: The Journal of Open, Distance and e-Learning, p. 9. Reprinted with permission.

Structure
The multi-disciplinary 5-State EU team ensured that the scope of the DIG-IT project could meet the structural requirements of funding, technical customisabiliy and scalablity, and ethical developmental considerations of its end-products prior to development of healthcare apps.

Establishing necessity for mobile healthcare apps
An extensive literature review identified the necessity for access to post-graduate online healthcare professional learning across EU states (Association for Medical Education in Europe, 2019; Corish, 2018; Daniel & Wolbrink, 2019). Continuing professional development is a process of career-long learning based on enhanced knowledge, expertise, competence, skills, performance, and adherence to standards that results in improved practice and patient safety (European Union, 2017; EU Executive Agency for Health and Consumers, 2013) Two primary goals of the Commission of the European Union were to provide accessible continuing professional development opportunities to healthcare practitioners and increase the quantity and quality of healthcare education at the point of patient contact by taking advantage of available digital technologies (European Commission Digital Education Action Plan, 2021).-2027 (European Union, 2021) Council of the European Union, 2017). The justification for this is that millions of European physicians and nurses must gain and maintain standards of excellence within the everchanging healthcare landscape (Eurostat, 2018(Eurostat, , 2020. Although headway had been made across the EU academic landscape in transitioning to online environments, the advent of pandemicrelated social distancing advanced some rapid technological support for online and mobile  Clendinneng et al., Cogent Education (2022) learning (Ali, 2020). Subsequently accessible, convenient approaches to education and training were in demand by students and professionals alike, within academic environments and for CPD. Chaotic pandemic conditions have also spurred an approach to integrated health promotion and care (World Health Organization Europe, 2021) that emphasized the professional obligation to keep up with the pace of technological and scientific changes.
To maintain excellence in practice in tandem with societal, economic, geopolitical, and environmental transformations, (Scott et al., 2017;Weglicki et al., 2015) healthcare practitioners assume a personal time and financial commitment to learning that spans the continuum of their practice in an unremitting cycle of upgrading evidence-based knowledge, skills, and competencies. Recent development and use of multimedia-enhanced, mobile content has enhanced the quality of healthcare education through provision of open, time-flexible options for practitioners via personal-use devices. This has fostered additional experiences where learners can take ownership of their learning (World Bank, 2020b).
Creating CPD for delivery on mobile devices for healthcare practitioners was a calculated decision. Mobile devices are the fastest growing educational technology, and these devices have increasingly permeated healthcare environments; 88% of physicians reported using mobile technology daily to deliver patient care (Mahajan, 2021). With portable communication and computing features allowing easy access and use, mobile devices have become integrated into clinical practice for such tasks as communicating between providers and patients, accessing health records, monitoring, and diagnosing patients remotely, and accessing medical education tools (Hitti et al., 2021). The advantages of mobile devices for rapid, lower error decision-making, improved data management and access to evidence-based information has ensured ubiquitous mobile uptake amongst healthcare providers making it a new occupational reality. Based on these trends, the DIG-IT team considered that appbased educational technology addressed upskilling and CPD in adaptable ways. Content can be customised and translated for varied practitioners; apps encourage reusability and promote scalability thus reducing design time and delivery costs across states and institutions.

Community
The EU-DEF framework describes a digital education environment that enables learners to work together to explore a significant question or create a meaningful project in a safe, participatory setting. This definition has been applied here to describe how the Digital Education and Timely Solutions team collaborated on developing the healthcare apps. The group consisted of experts who worked on aspects of the project relevant to their expertise. Thus the app project evolved through discussion, reflection, and collaboration mirroring effective learning and project management processes.

The process
Work on the mobile healthcare app development began in November 2019 with an educational needs assessment survey. This was developed and sent out by two nurse administrators from Saint James Hospital to their healthcare practitioners, primarily nurses, who identified potential evidence-based, patient safety related, educational topics beneficial across their six facilities located in Malta, Hungary, and Libya. Results from the survey were discussed with the subject matter expert, and two topics agreed upon for healthcare app development based on their projected breadth of relevance. Electrocardiogram Interpretation as a healthcare app choice was meant to benefit a wide audience of healthcare professionals with relevance to all nurses, nursing students, and junior physicians. Nurse Leadership and Management was chosen as an educational app because it is recognised and distinguished as pertinent in all healthcare systems. The International Council of Nurses' Nursing Now campaign substantiated this stance (International Council of Nurses, 2019). The Council accentuated and encouraged nurse leadership and management education as the cornerstone of world-wide health and social care systems, so consequently this topic has a global reach.
With the topics for development chosen, the team started planning and implementing the project. The app team consisted of a senior educational researcher/project lead, and several nurses: the two nurse administrators from Saint James Hospital, a nursing subject matter expert conversant with online design, a professor from the nursing faculty, University of Malta (UM), a nurse/translator from the Sapienza University of Rome, and two nurses who were eLearning experts from JAMK University of Applied Sciences, Finland. Program experts, a curriculum designer and program evaluator from UM, and four digital experts from the Software Engineering & Internet Technologies (SEIT) Lab, University of Cyprus, comprised the remainder of this experienced team.
Meeting together monthly on Zoom for virtual discussions and progress reports provided the foundation for these international members to develop a sense of identity and trust within the group which occurs when relationships advance, and members become valued within the community. The DIG-IT process of developing healthcare apps was thoughtful and participatory; questioning, clarifying, and visioning ensured valid, fair contribution of all participants, collaborative decision-making, and equitable distribution of deliverables.

Content, delivery and support
The next level in the EU-DEF addresses content, delivery, and learner support. The DIG-IT project team agreed that the subject matter expert would draft content for a 5-module ECG Interpretation app first followed by an 8-module Nurse Leadership and Management app. Development of both apps commenced with a literature review, setting the learning outcomes, and cataloguing relevant learning resources to align with them as the content of a digital education program must provide learners with the means to attain all required learning outcomes.
The healthcare apps were created using established asynchronous online pedagogies: andragogy; self-directed heutagogy; and problem-based learning (Liu et al., 2020) that were delivered in a mobile, micro-learning format. Contextual factors considered when and where healthcare practitioners need to retrieve information. Be it at point of patient care, or during brief interludes in their workday, they want learning experiences that are practical and immediately applicable. "New innovations enable learners to extend learning outside the boundaries of traditional learning institutions through informal and enriched learning experiences . . . " (Saykili, 2019, p. 1). Learning apps provided a flexible solution to engage anywhere there was a computational device and internet services. Grounding the content in theory created CPD activities that align with the transfer of knowledge, skills, and judgment into clinical practice.
Healthcare app content was organized into succinct segments or "chunks" that presented a comprehensive overview of information. Each digital education session began with a statement of three to five learning outcomes and an estimated amount of time required to complete the section. All modules presented objective information, interactive activities, a conclusion, takehome messages, and a self-evaluation exercise. As the modules' design was for the active, mobile learner, content was kept to concise, brief segments that could be viewed in short time frames. Information progressed logically scaffolding from basic principles to more complex concepts, and used a combination of text, graphics, audio, and animated multimedia instructional materials to explain difficult concepts (Hitti et al., 2021). This engagement provided learners with a sense of pacing and completion.
Once each draft app was completed, it was reviewed by nurses and curriculum designers at the University of Malta, the Sapienza University of Rome, and the Saint James Hospital team. Experts from each organisation made editorial, content, and configurational changes. For instance, a summary message for each module was added and terminology adjusted so it was appropriate for the EU multicultural, multilingual, contemporary healthcare workforce. When all team members were satisfied, the content was sent to an Italian nurse and a doctor/professor from La Sapienza University for translation into Italian. The translation and verification by external and internal reviewers ensured accuracy in both languages. Subsequently the SEIT Lab in Cyprus completed the programming and uploaded the apps to the project-digit.eu website where they were made available for pilot-beta testing.
Pilot project participants for the ECG app consisted of 10 nurses and physicians, who accessed the mobile learning activities by computer or mobile device through an open portal. Outcomes related to learner satisfaction were discerned through a summative feedback survey that elicited information on the framework variables: content, delivery, support, structure, community, and outcomes. Hospital staff feedback regarding the use of the ECG interpretation app was very favourable; all users found the app's content relevant and easy to use. Participants commented: "Course was well designed with user friendly approach, informative, easy to follow, and included videos . . . it gives a very clear picture of how the heart chambers contracts in different arrythmia scenarios". "Modules will be of great assistance for health care professionals of all levels, but, especially for those who work in an emergency setting." "Modules really helped me to brush-up on cardiac arrhythmias ECG interpretations, so . . . these will be helpful for a beginner." Some feedback to developers resulted in minor improvements to the app quizzes and all beta tester learners who successfully completed learning activities received a certificate of achievement for their files and institutional tracking.
The ECG app was the first to go live in November 2020 and to date of over 400 participants have successfully completed and been awarded certificates for this CPD activity. The Nurse Leadership and Management followed with a June 2021 launch date. To date the number of participants who have completed this module is low. The team presumes this is due to the summer launch with healthcare students and worker vacation, and plan to promote uptake in the fall. The team is hopeful that these numbers will increase exponentially now that healthcare workers are experiencing more free time to engage in such activities.

Intentional use of technology
All digital educational development requires an advanced understanding of appropriate technology. Educators/developers must have information and resources on the nature and implications of technology tool options. For example, Steele's (2014) Intentional Use of Technology Model, is a tool guiding informed, calculated decision-making on the use of technology and interactive media that supports learning and development (Radich, 2013;Steele, 2014). Steele proposed that three broad categories overlap with one-another: enterprise level systems, and communication and Webbased tools. The category of enterprise refers to the use of a system-wide approach to computing power that services both the institution and students. Communication tools should focus on student to student, advisor to student, and institution to student interactions. Finally, web-based tools centre on the provision of information using an assortment of media. Steele's method and model aligned with the EU-DEF and was chosen for app development. Selective and relevant use of technology by the SEIT Lab team resulted in educational solutions that adhered to ethical principles that are respectful and confidential, are founded on pedagogical, supportive approaches, include valid, fair assessments, and ensure smooth, flexible, convenient digital study paths for comfortable learner usability.

Usability
The SEIT team also had to address usability. The International Standardization Organization (ISO) 9241-210 (2019) defined usability as "the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" (p. 3.13). Usability ensures that user interfaces are easy to use and support learners in performing their tasks. Consequently, usability is more concerned with the tasks that users must perform and their level of accomplishment. The EU-DEF suggests that at minimum usability levels must consider that: information on the site must be kept up-to-date with no dead ends or stale links; the learning environment must be easy to navigate; and careful organisation of the learning resource should minimise the number of steps or "clicks" that learners need to perform. These aspects were considered in the design and development of the DIG-IT healthcare apps. Information is kept updated, and all links lead to appropriate destinations and resources. A simple navigation structure has been employed enabling users to progress easily between the learning content, outcomes, and questions for each module in a study-unit, session, course, or program. See Figure 2 ECG App Simple Navigation Structure.
Moreover, to enhance usability a standard design was adopted using specific icons to distinguish reading, watching, applying, and testing components throughout the learning content. This helps learners recognise the type of cognitive and stimulation activity that they will engage in with specific content. See Figure 3. ECG App Standard Icons Distinguish Learning Activity

Accessibility
In conjunction with usability comes accessibility. The ISO 9241-210 (2019) defined accessibility for interactive systems as "usability of a product/service by people with the widest range of capabilities". Viewing the relationship between usability and accessibility in depth increases clarity, since their overlap creates misconceptions. Yesilada et al. (2012) distinguished several views of this relationship. The first is that problems for accessibility and usability are distinct, meaning that problems experienced by people with disabilities and people without are different. Second is that accessibility problems are a subset of usability problems. Third "universal usability" encompasses both usability and accessibility problems. This infers that the scope of usability is broadened to address problems disabled people experience. The fact that there is debate on a common understanding of accessibility makes it more difficult to define this relationship. To harmonise an understanding of accessibility via the comparison of different definitions, Yesilada et al. concluded that web accessibility means that people with disabilities can use the web; they can perceive, understand, navigate, and interact, and contribute to it (ISO/IEC 40500, 2012). Universal accessibility from the perspective of access to everyone, irrespective of human abilities is still a work in progress for the healthcare app users, with the aim to provide this environment at the end of the project.
In line with the EU-DEF, accessibility points more towards offering access to the learning content. It focuses on the selective and relevant use of technology to support smooth, flexible, and convenient digital study paths for learners. These educational solutions meet future competence needs.
The healthcare apps provide these qualities enabling learners to access and interact with the learning resources on all operating systems at their chosen time and environment. The use of different screen resolutions offers options to use various and preferred devices for learning and the design and development of the apps include specific strategies to maximise the learning experience and to stimulate learner interest. For instance, to facilitate a wider cross-cultural distribution in the EU, Italian-translations of the English version of the app courses were integrated in the tool to specifically include at least two partnering countries, Italy, and Slovenia where 20% of the population speaks Italian. Translation of the ECG Interpretation app took place in May 2020 and was released on the DIG-IT website in July 2020, and the Nurse Leadership and Management course was translated into Italian in September 2020 and released in March 2021. This decision was projected to enable a broader uptake and awareness of the project and help reach the DIG-IT team's initial target of 1000 healthcare professionals-medical and nursing students and residents at several international hospitals and academic facilities.

Segmenting content
To avoid overloading the learner with information all at once, each course is divided into four distinct sections: Introduction, Learning Outcomes, Learning Activities, and Modules. In the Introductory section learners are provided with a clear roadmap of the aims of the course and the modules included within, as well as the expectations and estimated hours needed to complete the entire course. The different types of activities are clearly presented followed by the learning outcomes and score learners must attain to "pass" and move on to the next module.
The environment of each app was designed in a minimal, simple-to-use way. The web technologies used for the development such as PHP, JS, HTML5, provided the means for creating interactive software tools. As such, the course is divided into different tabs, that carry the sections, and enable the interchange of content displayed based on the learners' navigation selection. Learners can only view the modules already completed or the current module they are working on. Modules, like the courses, are structured in three distinct sections maintaining a consistent style throughout the tool. This simple design avoided confusing learners with complex web design interfaces and interactive tool facilitate their knowledge path. Having the apps housed and disseminated on the DIG-IT site enables global access and awareness. The convenient accessibility on an iPad or iPhone, allows the content to be referred to at the patient point-of-care or in any healthcare setting.

Technical/media training and technical support
Well known learning environments and learning management systems provide a complex interface with all required functionality for learners and instructors/tutors. The SEIT Lab team designed and developed the two DIG-IT apps addressing the needs of functionality and user interface by using the WordPress Content Management System as the basis of the development. WordPress is open source, allowing developers to add functionality on top of it and is used by 33.1% of all websites today (Web3Techs, 2021). This content management system performed well when it underwent comparative performance analysis in Patel et al.'s (2011) research. Today this free software is touted as flexible, customizable, and user-friendly and has become the most popular content management system used to build, modify, and power websites on the internet (Domantas, 2021). As WordPress is familiar to many learners it eliminates the learning curve that would be needed with a new learning environment. Thus, based on this design decision, no learner orientation to the environment was needed, however creation of a user manual is planned.
Beta testing was the final stage of app development. First, a testing phase was organised within the team of University of Cyprus, followed by an internal testing within the DIG-IT project team. The tools were used extensively, and bugs and errors were identified and addressed. During the pilot testing, learners provided feedback and suggestions for the tools. The collected feedback was examined, and changes applied to the courses. Technical support remains ongoing to all course learners; they are prompted to contact the team for any problem they face, whether administrative, technical, or educational, however there has been little technical support required.

Delivery
According to the EU-DEF, content must align with the learners' development and in healthcare must keep with uppermost professional and disciplinary practices. The healthcare apps incorporate pedagogical strategies that engage learners in higher level learning by using a variety of instructional strategies and activities, applying a wide range of learning models to accommodate different learning styles and, eventually, maximising and leveraging the experience of each participant. To ensure an increase in learner knowledge and skills in both apps, after each module, learners are required to take and score 100% on a short multiple-choice quiz before moving to the next, more complex module. Transfer of knowledge to clinical environments is encouraged through learner participation in realistic case study scenarios that make connections between the information they absorb and patient care.

Outcomes
Academic staff were dynamically involved in the app development, so their satisfaction was assured prior to launch. After the launch of the ECG Interpretation app, informal user feedback was obtained from nurses and physicians. The overarching comments were that these users appreciated having their voices heard up front regarding topics for app development. Nurses were particularly pleased that the app content was relevant, of high quality, and addressed the identified learning needs of ECG interpretation and nurse leadership and management development. Feedback also confirmed that providing free, convenient, carefully planned, accessible training resulted in benefits for hospital staff. As mobile learning content was designed specifically for personal convenience, nurses and physicians appreciated the flexibility and self-direction to learn while considering their busy work commitments.

Recommendations
Current users and personnel involved in the future evolution of these apps must respect the fact that the applications were developed across educational and healthcare entities based on a European context including mind-set, organisational cultures, work practices, and content terminology. Although the European context may not be fully congruent elsewhere this limitation holds less significance against the backdrop of the phenomena of globalisation and health care staff mobility.
As with many healthcare educational initiatives, the main goal is to improve patient care and outcomes. This implies that there is no attrition of learner knowledge over time and there is an active transfer of knowledge and skills to the clinical setting. A recommendation for these healthcare apps is to create specific evaluations to determine the extent to which knowledge was acquired, retained, and transferred to the clinical setting for learners who completed their certificates. This would provide more definitive results on the value of the apps and their organizational impacts.
The challenges of future healthcare education and app development relies on creative teamwork and collaboration. The DIG-IT team was a "dream team" in that there were members from healthcare, academia, and technology who were appropriately resourced with time and funding to accomplish specific research and outputs. As technologies evolve, people are re-imagining healthcare delivery and education. Healthcare education will be influenced by artificial intelligence, telemedicine, virtual, and augmented reality. Patient-centered health advice and wellness measures will intersect with individualized care initiatives that are driven by the digital technology industry and patient/practitionner initiatives; thus, development teams will consist of a variety of stakeholders providing input into educational applications. As the future of healthcare educational delivery will be motile, the reminder for developers is to use an evidence-based approach to design, deliver, evaluate, and disseminate research expertise.

Conclusion
The DIG-IT project facilitated the collaboration between university academics, clinical educators, industry partners, healthcare professionals, and technology experts over 5 EU states, prior to and during the pandemic, to pioneer the creation of digital resources for mobile healthcare learning. This transnational project allowed team members to learn with and from one another about mobile digital education. This type of teamwork led to the responsive, efficient creation of worldclass educational products. Collaboration encouraged reusability, adaptability, and scalability of digital education and reduced the time to diffuse innovation and change. For healthcare professionals, mobile learning promotes participation, shortens studying times, creates versatile study paths, and ultimately, secures and maintains better employment and patient healthcare outcomes. The EU-DEF provided a quality standard to guide the design of the mobile online healthcare applications. Developers reflected on and incorporated the essential variables and sub variables identified and operationalised in the framework. The DIG-IT team encourages others to take advantage of the EU-DEF and follow a similar trajectory for developing digital education products.