University of Birmingham Critical analysis of a higher education benchmark via fuzzy logic

A graduate pro ﬁ le benchmark developed by a higher education organisation is subject to scrutiny. This occurs through the numerical parameterisation of text collected through surveys and the use of basic concepts drawn from Information Technology. The process uses data processing, analysis and evaluation to show the degree to which established an academic practice at the host institution correlates to pre-con-ceived professional outputs. The paper discusses standard practices including traditional and innovative teaching meth-ods, in the light of achieved results. To conclude that post-graduate students require further engagement with the industry and exposure to the public whilst showing that the scienti ﬁ c and technical components of the subject courses are highly rated by stakeholders. Mitigation measures for the identi ﬁ ed gaps include further curriculum development and inter-disciplinary work, reinforced with industrial liaison plus engagement with learned organisations and the public, via educational enhancement activities.


Introduction
The national framework for higher education qualifications (FHEQ) in the United Kingdom exists since 2001. It has its origins in the Dearing Report (The National Committee of Inquiry into Higher Education, 1997) and is a reference point for educational providers, to maintain academic standards, inform international comparability, ensure competitiveness, and facilitate student and graduate mobility (QAA, 2008). Its basic premise is that qualifications should be awarded based on the achievement of outcomes as opposed to years of study. For some subject areas (QAA, 2015) suggests that higher education providers need to consider additional reference points to enable incorporating requirements set out by professional, regulatory and statutory bodies (PSRBs), national educational standards and industry or employer expectations. In terms of Master's degree in Engineering (QAA, 2015) defines a series of attributes seen as necessary to face those challenges threatening modern societies, although recognises the freedom of education providers to develop and assess their own curriculum. HE institutions have thus the ability to scrutinise views from stakeholders, namely, prospective students, schools, parents and employers (QAA, 2008). In that sense, the School of Engineering (SoE) at the University of Birmingham (UoB) has shaped a benchmark profile named Birmingham Engineering Graduate. This profile comprises attributes spanning from subject specified knowledge to creative thinking, passing through enterprising and transformative thinking, global and cultural awareness, leadership, and ethics (University of Birmingham, 2016). The graduate profile is consistent with the QAA benchmark for Master's degree characteristics (QAA, 2010) and intends to satisfy needs identified by affiliated academic and industrialists. Attributes of the Birmingham Engineering Graduate are compatible with benchmark statements established by other HE institutions. Although to the author's best knowledge, shaping specific benchmark profiles for graduates is not shared practice across the sector. In such scenario is required to translate the descriptors developed at UoB into outputs that measure and quantify students' performance and satisfaction, employability, and degree classification, amongst others parameters. The created benchmark could also inform PSRBs about academic practice and development. Those instruments for monitoring the quality and effectiveness of curricula often reveal areas for improvement. The present study critically analyses the extent to which academic practice at SoE enables attributes of the Birmingham Engineering Graduate. To that end, opinions from students, visiting lecturers, academics, and alumni of two masters engineering programmes formed a database. Answers to questionnaires stored in the database translated to rational numbers through parameterisation of the data. The numerical information then passed through a filter based on fuzzy sets. Fuzzy logic started developing in the 1960 to enable scientists processing information without having to classify it as TRUE or FALSE. Degrees of TRUTH made then possible to manipulate information that carries degrees of subjectivity (Zadeh, 1985). This theory has influenced various fields of science including but not restricted to control theory and artificial intelligence. Its formulation enables to identify patterns, features, and behaviours, within data through suitable approximations such as those reported in Takagi and Sugeno (1985), Murray Smith and Johansen (1997), Juuso (1999), Chih-Hsiu andChin-Shyurng (2002), and Gobi and Pedrycz (2005). More recently, covering education specific subjects we have seen studies by Sobrino (2013), Soares, Ribeiro, Autran, Machado, and Jusan (2016), Garrido (2018), and Sasmoko et al. (2019). Sobrino (2013), explored forms to introduce fuzzy logic to students by using examples of everyday activities. That investigation illustrates the complexity of mapping subjective information, e.g. vague language, into objective results, that can inform decisionmaking processes. Soares et al. (2016) established an intelligent tutoring system that reinforces natural learning while acknowledging the various cognitive layers that conform individual learning. Garrido (2018) applies fuzzy logic as an attempt to sophisticated machine learning. Their goal is to access subtle reasoning that characterise humans and enable artificial intelligence to mimic specific behaviours such as inductive and deductive reasoning. Sasmoko et al. (2019) implemented a self-assessment exercise for teachers. In their model, teaching quality involves features such as Openness, Clearness, Enthusiasm, Teaching Methods, Feedback, and Commitment. They progressed these into fuzzy sets fed with field data obtained from a student sample. The output data constituted an indicative of teaching quality with specification of areas for improvement. The fuzzy approach thus deviates from deterministic methods that use or produce single-numbered sources and moves away from stochastic methods based on single-valued algorithms. This enhances the scope of problems that scientist can tackle and maximises the potential that artificial intelligence can provide.
The mathematical approach outlined above acts as object in the present investigation. The subject is the target profile for engineering graduates shaped at the University of Birmingham. The present investigation thus deals with the critical assessment of the engineering graduate profile by collecting, processing and interpreting numerical information mapped from statements originally captured through surveys. Data processed through fuzzy logic have enabled to critically assessing the correlation between academic practice and attributes of Masters' engineering graduates. The analysis also helped to identify areas of improvement and good practices for potential dissemination. The novelty of this approach thus resides in the use of fuzzy logic to framing the study and to systematically attempt to uncover areas for enhancement in teaching of industry-ready students.

The Birmingham Engineering Graduate
Features of the Birmingham Graduate develop progressively through institutional and School-specific educational experiences. These also derive from opportunities including curricular and extracurricular activities directed to enrich and diversify the education portfolio at Birmingham (University of Birmingham, 2016). The School of Engineering is committed to deliver these principles locally through establishing a subject-specific platform that aligns with the institutional ethos and mission. This vision is consistent with the descriptor of characteristics of Master's degrees established in (QAA, 2015). Although the QAA descriptor does not form part of the Academic Infrastructure reflected in the Framework for Higher Education Qualifications in England, Wales and Northern Ireland (QAA, 2008), is indicative of the educational output for postgraduate taught engineering students. The QAA descriptor is therefore flexible. It enables educational providers to design, deliver, and evaluate their programmes of study on regular basis while leaving room for innovation and enhancement. Table 1 shows characteristics of the Birmingham Engineering Graduate (Cooke & Hawwash, 2017) and the QAA Descriptor while Table 2 correlates teaching methods and techniques that support attributes of the subject educational benchmark. Table 1 outlines the correlation between descriptors. These address three core elements in engineering, namely, scientific knowledge, mathematics, and realisation. Scientific knowledge underpins engineering while mathematics enables translating parameterisation into engineering solutions. Realisation is core to our discipline because forces engineering solutions into tangible property or invention, often attached to commercial or social value (QAA, 2015). The constructive alignment depicted in Table 2 promotes attributes of the Engineering Graduate. The adopted teaching and learning methods fit into the UK Professional Standards Framework developed by The Higher Education Academy and on behalf of the UK higher education sector, Guild HE and Universities UK (HEA, 2011). That framework spans from the design and planning of learning activities, developing effective learning environments and engaging with the wider engineering community, to the acknowledgement of the context in which we operate and the respect of individual learners and learning communities. This evidently requires awareness and understanding of cognitive process that lead to learning and how these can be further stimulated through applied technology while enforced through methodical quality assurance procedures.
In this context, the challenge of the educational provider is to creating mechanisms to progressively building students' profiles into the developed benchmark scheme. That process requires constant reflection on the quality of academic practice to make adjustments or improvements as required. The present study intends to inform that process through the critical assessment of academic practice with respect to the Birmingham Engineering Graduate benchmark. The study consists of analysing a database containing the opinions expressed by students, alumni, and staff. The information collected then goes through a filter that enables cross-correlating benchmark attributes with personal experiences manifested by interviewees. The results obtained provide the basis to conclude with some general remarks.

Data collection
The assessment of acquired attributes by engineering graduates uses as context two postgraduate taught engineering programmes. These are fully accredited programmes by the Joint Board of Moderators (JBM). This organisation brings together the Institution of Civil Engineers (ICE), the Institution of Structural Engineers (IStructE), the Chartered Institution of Highways and Transportation (CIHT), and Seek to achieve sustainable solutions to problems and have strategies for being creative, innovative and overcoming difficulties by employing their skills, knowledge and understanding in a flexible manner Be skilled at solving problems by applying their numerical, computational, analytical and technical skills, using appropriate tools Knowledge, understanding, and application of scientific and mathematical skills Research skills for effectively identify sources of information, implement techniques to process and analyse information, and provide constructive criticism to other's investigations. Action-oriented skills: taking initiative, coping with risk and working with others.
Be risk, cost and value-conscious, and aware of their ethical, social, cultural, environmental, health and safety, and wider professional responsibilities Be professional in their outlook, be capable of team working, be effective communicators, and be able to exercise responsibility and sound management approaches Collaboration: ability to work with professional and technicians from other disciplines. Communication skills to be transmit complex or abstract ideas to technical and non-technical audiences.
Appreciate the global dimensions of engineering, commerce and communication, be able to formulate and operate within appropriate codes of conduct, when faced with an ethical issue Awareness of economic, environmental, legal, social, health, safety and ethical issues. Public good to address grand and regional issues. Additional general skills including planning self-learning and exercising initiative and responsibility. Be pragmatic, taking a systematic approach and the logical and practical steps necessary for, often complex, concepts to become reality Resourcefulness, demonstrated through self-awareness and self-efficiency, motivation and perseverance, mobilising resource, financial and economic literacy, mobilising others. Critical thinking for recognising shared problem spaces and establishing connections with others to tackle them. Ideas and opportunities: spotting them, developing vision and valuing ideas.
Be familiar with the nature of business and enterprise in the creation of economic and social value the Institute of Highway Engineers (IHE). The average student population in these MSc programmes totalised 50 in the academic session 2017-18. To determine the size of a sample that is representative of the cohorts, Cochran's sample size formulae apply. Equation (1a) provides the initial estimation of the sample with a confidence level of 95%. Bartlett, Kotrlik and Barlett, Kotrlik, and Higgins (2001) suggest using a tabulated t-value of 1.96, corresponding to a population of 60. In this equation, the variable s and d are the standard deviation and target margin of error, respectively. The initial values of these parameters are correspondingly 30% and 10%.
Since the real sample size is 50 and not 60, Cochran's correction formula becomes relevant. Equation (1b) shows that an adjusted sample size of 21 students provides 5% uncertainty, which satisfies the purpose of the investigation. Following, a survey was prepared to capture students' views on the suitability of the educational curriculum imparted at SoE with regard to attributes of the Birmingham Engineering Graduate. The Descriptors shown in Table 1 separate into the 16 attributes listed in Table 3 ( Dvornik, 2018). The attributes were then included in a survey alongside a numerical scale ranging between 0 and 10, for grading.
The scale shown in Table 4 intended to give flexibility for individuals to express their opinion about the extent to which the educational approach at SoE helps students to achieve the Birmingham Engineering Graduate attributes. Figure 1 shows the trend of scores captured by individual student responses per attribute whilst Figure 2 shows the corresponding average ratings as extracted from raw data. Awareness of all aspects 5 Engineering practice 6 Additional general skills. 7 Research skilled 8 Ideas and opportunities 9 Resourcefulness 10 Action-oriented skills 11 Collaboration 12 Critical thinking while collaborating 13 Deal with uncertainty while collaborating 14 Technical skills. 15 Modern industry skills 16 Public good. Unsatisfactory level of coverage 5 Barely satisfactory level of coverage 6 Satisfactory level of coverage 7 Decent coverage 8 Good coverage 9 Excellent coverage 10 Skill is fully covered The average rating depicted in Figure 2 is of 6.95 across attributes, with a standard deviation of 0.44. The average standard deviation across individual opinions per attribute, i.e. the average amplitude of the curve plotted in Figure 1, is 1.98: while the corresponding maximum value was of 2.7attached to Engineering Practice, and the corresponding minimum standard deviation related to Engineering Analysis Skills with 1.65.
As stated above, the survey circulated among students also reached alumni (5), staff (1) and visiting lecturers (VLs) (2). However these secondary populations are smaller, their input enriched the database. Figure 3 shows the average rating per attribute provided by alumni while Figure 4 shows the corresponding results obtained from staff and VLs.  5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20   The average scores provided by all three groups across attributes are G 1 = 6.95 (students), G 2 = 6.94 (graduates), and G 3 = 6.73 (staff and visiting lecturers). Notwithstanding the consistency of these figures, there are differences in the opinions expressed on by these groups. For example, G 3 gave a score of 8.67 to both Scientific and Mathematical Skills and Engineering Analysis Skills, whereas G 2 rated the same pair with 7.2 and 7.4, respectively, and G 1 graded them with 6.67 and 7.05. Thus, opinions on these two differ on average in 18.7% (G 3 , G 2 ) and 26.4% (G 3 , G 1 ). It is noted that, G 3 gave an average score of 5.11 across three attributes, namely Collaboration, Critical Thinking while Collaborating, and Deal with Uncertainty while Collaborating, whereas G 2 and G 1 rated the same three attributes with an average of 6.7 (33% higher with respect to G 3 ) and 7.05 (37.9% higher with respect to G 3 ), respectively. Figure 5     an overview of the scatter observed across responses obtained from the three groups, per attribute. The results above already enable to draw some partial conclusions regarding the effectiveness of the engineering curriculum and academic practice at SoE. However, such analysis would remain incomplete if we ignore the intrinsic correlation across attributes. We therefore hypothesise that enabling soft boundaries across attributes, would result in a more realistic appreciation of education practice. Also recognising that the subject benchmark spans in 16 directions, which by no means are disjointed. To illustrate this, let us take the example of Design Skills which whilst underpinned by Scientific and Mathematical Skills necessarily relates to Analysis Skills. A second example could include Modern Industry Skills, which could be seen as close to Collaboration but not unrelated to Ideas and Opportunities and Public Good. The list of examples could go on. Average scores accumulated by each one of the target attributes should therefore count to determine the net score of any other attribute.
The following section describes the mathematical framework established to re-distributing scores to reflect the relationship amongst attributes.

Mathematical model for data processing and results
Past research shows how Fuzzy Logic is applicable to a broad range of scientific disciplines. Education science is not the exception while the nature of the discipline does require any particular modification to the theory. The basic steps to develop a fuzzy model requires identifying input data and desired output information. Fuzzy Awareness of all aspects. Engineering practice.
Additional general skills. Research skilled.
Ideas and opportunities. Resourcefulness.
Critical thinking while collaborating.
Deal with uncertainty while collaborating.
Technical skills.
Modern industry skills. Public good.
Staff -VLs Graduates Students theory lies in between these, hence it is a mechanism to filtering data with the particular characteristic that the input information is not deterministic, i.e. does not need to be in numerical form, originally. Such is the case of information collected in text form, for example, through a survey. The basic steps to develop a fuzzy model thus require formulating sets that (i) capture input information and (ii) provide specific outputs that the user wishes to investigate. Fuzzy sets can be graphically represented but require a scale, typically ranging between the interval 0-1, but the scale could be different. Each set thus represent specific input, for example, it could be the features of an object such as shape, size, consistency, mass, etc. These sets would therefore capture controlling parameters of the system's response. For example, IF the object is oval AND large AND [solid OR non-porous] AND heavy, THEN, its impact on a surface, e.g. made of glass, WILL cause damage. In this context, the reader would identify other important element of fuzzy modelling that is the existence of fuzzy rules. The user develops those rules and usually refer to logical statements drawn from classical probability. In the example above, the features of the object represent the input data, the IF-THEN is the rule, and the conclusion of the statement represent the output. The latter can be refined to include degrees of damage. It is important to note that a fuzzy model maps input to output parameters through a numerical processing, hence each characteristic on either side of the system (input/output) needs a numerical measure added to it.
In line with the above, fuzzy sets can be seen as two-dimensional geometrical regions that can overlap therefore generate memberships across clusters. In this investigation, fuzzy sets represent the Birmingham Graduate attributes (A k ) listed in Table 3 where 1 ≤ k ≤ 16. These sets characterise by having a partial score, granted to them during the survey, and certain overlapping with other sets. Figure 6 illustrates the proposed scheme.
The vertical axis in Figure 6(a) measures the original score [0-10] while shaded areas illustrate degrees of membership across sets. Figure 6(b) represents a hypothetic coherent structure derived from the correlation amongst attributeshere represented as abstract ensemble within a common region. Cross correlation of sets can therefore be strong or weak and include one or more attribute(s).
Membership values derive from the original scores. Interviewees established such correlation unintentionally but based on notion and experience. For example, according to the data gathered they correlate Research Skills (A 7 ) with Analysis Skills (A 2 ) more strongly than with Critical Thinking wile Collaborating (A 12 ): the processing of raw data yielded membership values of μ 7;2 ¼ 0:733 and μ 7;12 ¼ 0:437, respectively. They see also Public Good more strongly correlated with Modern Industry Skills (μ 16;15 ¼ 0:829) and Ideas and Opportunities (μ 16;8 ¼ 0:703) than with Scientific and Mathematical Skills (μ 16;1 ¼ 0:32). Noting that, the Birmingham Engineering Graduate and QAA Descriptors do not correlate attributes. Membership values in the present study are therefore intrinsic to the opinions gathered through the survey. These reveal additional features of the database that escaped from the initial scrutiny shown in Figures 1-4. Table 5 shows the cross-correlation matrix, now referred to as membership values, extracted from the raw database. Membership values enable re-distributing scores across fuzzy sets. According to this, scores originally assigned to attributes trespass limits across sets that become permeable. The final score or grade assigned to each attribute is therefore determined with Equation (2).
In Equation (2), Ψ k is the final score of the k-th attribute, Ψ k;0 is the grade inferred from raw data while μ i;k ¼ μ k;i represents the membership value identified between i-th and k-th fuzzy sets. This scheme enables attributes to increase or decrease their score as a function of the identified synergies. Is possible to express Equation (2) in matrix form as follows, where Ψ F contains final scores (Ψ k ) after re-distribution, Ψ 0 is the vector of scores extracted from raw data, Ψ T D is the transpose of the matrix of differences amongst rows of Ψ 0the components of Ψ D result from subtracting the score of the attribute marked by the column label from that of the row label. M is the matrix with membership values,  and Diag is an operator that extracts those elements in the main diagonal of Ψ T D M is a vector of size N. To check the sensitivity of the results to the opinions expressed by either group, students (G 1 ), graduates (G 2 ), staff and visiting lecturers (G 3 ), is possible to weighing scores before combining them and passing them to the filter Equation (2) or Equation (3). Equation (4) establishes the combination rule to determine Ψ 0 .
where α; β; γ are weighing values whereas Ψ j;k;0 represents the score originally given by the jth subject group G 1 , G 2 , or G 3 , to the kth attribute. Table 6 shows the results of re-distributing scores subject to arbitrary weighing values. Figure 7 shows the results of the analysis. The vertical axis represents the final score within the scale 0-10 while each bar within the neighbourhood of anyone attribute relates to the weighing criteria given in Table 6. The results show little scatter derived from the weighing process and higher uniformity of scores across the board with respect to the values shown in Figure 4. The filtering of the raw data however did not modify the average score of 6.87 estimated across all attributes as per preliminary results depicted in Figure 5. The standard deviation of final scores with and without allowing re-distribution is 0.28 and 2.68, respectively.
According to these results, Research, Additional General Skills and Scientific and Mathematical Skills are at the top of the range with scores ranging between 7.2 and 7.3. At the bottom of the scale lies Industry Skills and Dealing with Uncertainty while Collaborating and Public Good with scores ranging between 6.4 and 6.6. Table 7 shows the overall scale of final scores and the grade range.
The range of scores in Table 7 represents 9.2% of the overall scale. Notwithstanding the relatively little scatter observed, the result shows that those attributes involving external instances such as industry and public [A 14 -A 16 ] have the lowest relative score. Attributes involving collaboration [A 10 -A 13 ] appear in the middle bottom range while Table 6. Final scores per attribute considering weighing factors and re-distribution.
practical and technical aspects of the discipline [A 5 -A 9 ] fall in the middle top range. Scientific core knowledge and research [A 1 -A 4 ] dominate within the scale. The results of the analysis provide elements to reflect on the existing curriculum at SoE in light of accreditation requirements. This needs correlating the attributes expressed in keywords listed in Table 3 with both the Birmingham Engineering Graduate benchmark and QAA Descriptor. Table 8 shows this mapping and the equivalent QAA scores.
The correlation simply derived from averaging the relevant scores that fall within the neighbourhood of QAA Descriptors. This pragmatic approach would otherwise follow a detailed analysis to determine weighting values or a direct equivalence between descriptors. The complexity of such task is not within the scope of the present investigation, although is worth to point out that the lack of such direct equivalence is consistent with the general principles established in QAA (2010) regarding Master's degree characteristics. Those reference points seek to reflect and accommodate the range of programme types, including aims, mode of delivery, methods of teaching, Public good to address grand and regional issues.

A14: 6.64
Additional general skills including planning self-learning and exercising initiative and responsibility.

A2: 7.25
Be pragmatic, taking a systematic approach and the logical and practical steps necessary for, often complex, concepts to become reality QAA6: 6.96 Resourcefulness, demonstrated through self-awareness and selfefficiency, motivation and perseverance, mobilising resource, financial and economic literacy, mobilising others.

A7: 6.96
Critical thinking for recognising shared problem spaces and establishing connections with others to tackle them.
A9: 6.83 A12: 6.66 Be familiar with the nature of business and enterprise in the creation of economic and social value QAA7: 6.75 learning and assessment, and intended outcomes, rather than to prescribe any particular model (QAA, 2010). The link between the analysis results and the standing QAA governing curriculum reveals higher uniformity in the distribution of scores. The standard deviation directly established from the scores derived from the study yields 0.28 while the so-called QAA scores report 0.10. As expected, the mean value of either set equals 6.88, i.e. the mapping did not modify the average score across descriptors. The inferred performance linked to the QAA Subject Benchmark Statement (QAA, 2015) suggest areas of improvement in subjects such as business and enterprising and the creation of economic and social value. At the top of the scale, i.e. 6.9 < QAA Score ≤ 7.00 we find numerical, computational, analytical and technical skills, team work and communication, the adoption of logical and systematic approaches to translate theory into practice, and the appreciation of the global dimensions of Engineering. In the middle range we have creativity and innovation, be risk, cost and value-conscious. This brief overview of equivalences suggests that while there is consistency between descriptors, those related to the Birmingham Engineering Graduate appear more segregated or specific while the QAA requirements implicitly recognise higher interdependence of skills achieved by engineering graduates.

Final discussion
The results of the study derive from critically confronting a higher education benchmark with an objective measure of academic practice. The subject engineering curriculum covering core subjects within the discipline spanning from analysis and design to sustainable development, passing through business management and complemented with the strengthening of core skills. The delivery of the MSc courses incorporates innovation in the form of lecture recording to make more flexible course schedules for part timers and students with disabilities; formative assessments to enable the alignment between module learning outcomes, teaching delivery and exams; re-design of course schedules for students to undertake self-learning, direct interaction with staff, and group work, at specific time-periods. Internal and external instances to the School of Engineering also oversee the elimination of cultural barriers, H&S and welfare issues, incorporating schemes going along and transverse to students and staff existing committees. In addition, industrial engagement occurs via research projects and formal teaching involving industrial partners while extra-curricular activities take the form of hands-on activities inside and outside the University campus. From the analysis presented in the study, it transpires that postgraduate students enrolled in the subject programmes require further exposition to public and industrial activities. The way forward, therefore, seems to try to expand elements of the current undergraduate curriculum at the SoE that promote inter-disciplinary work across students from three engineering disciplines and with students with different backgrounds drawn from external institutions. MSc programmes can, therefore, promote further curriculum development to allow interdisciplinary interaction. The initial steps towards this change already occurred via the creation of a common dissertation module that includes students from two other Departments, namely, Mechanical and Electrical Engineering. This could enable optimised learning environments to fit purpose. There is also space to intensify industrial engagement. The SoE and its Industrial Advisory Board could re-develop educational enhancement activities to increase awareness of public needs and requirements that are on demand in Civil Engineering. Further educational enhancement could bring in PSRBs to acknowledge more explicitly the implications of core values and skills in the professional practice. Those actions could potentially reduce the gap observed in the final scores amongst attributes and configure an optimised profile of modern engineering graduates.

Disclosure statement
No potential conflict of interest was reported by the author.