Exploiting data from safety investigations and processes to assess performance of safety management aspects

Abstract This paper presents an alternative way to use records from safety investigations as a means to support the evaluation of safety management (SM) aspects. Datasets from safety investigation reports and progress records of an aviation organization were analyzed with the scope of assessing safety management’s role, speed of safety communication, timeliness of safety investigation processes and realization of safety recommendations, and the extent of convergence among SM and investigation teams. The results suggested an interfering role of the safety department, severe delays in safety investigations, timely implementation of recommendations, quick dissemination of investigation reports to the end-users, and a low ratio of investigation team recommendations included in the final safety investigation reports. The results were attributed to non-scalable safety investigation procedures, ineffective resource management, lack of consistent bidirectional communication, lack of investigators’ awareness about the overall organizational context, and a weak commitment of other departments to the realization of safety recommendations. The set of metrics and the combination of quantitative and qualitative methods presented in this paper can support organizations to the transition towards a performance-based evaluation of safety management.


Introduction
Quality assurance and performance measurement are tightly related. The latter furnishes the former with data necessary to depict organizational performance over time and conduct benchmarking studies amongst departments, organizations, industry sectors etc. (Stapenhurst, 2009). Organizations measure performance in order to identify success, reveal potentially weak internal processes, support decisions with facts and monitor implementation of improvement actions.
Regarding safety management (SM), a recent study concluded that the aviation sector organizations evaluate their performance mainly through compliance-based audits, safety culture surveys and metrics which focus mainly on event outcomes and secondarily on SM processes (Kaspers, Karanikas, Roelen, Piric, & de Boer, 2016). Although aviation regulators and international bodies have pointed out the need for introducing a performance-based scheme for SM assessment (e.g. EASA, 2014; ICAO, 2013b), few guidelines have been published to fulfil this requirement.
Accident and incident investigations are part of safety assurance and have played an important role in revealing organizational deficiencies and generating recommendations for improvements.
Organizations and authorities use results from safety investigations to calculate event rates per severity category and frequencies of causal factors as indications of their safety performance (e.g. Airbus, 2016;Boeing, 2016;EASA, 2016;HSE, 2016). However, to date there has been no literature suggesting how organizations can exploit data from safety investigation reports and processes as a means to develop metrics for evaluating whether their SM is effectively implemented.
This study exploited data from accident and incident investigations of an aviation organization and illustrated the degree to which it effectively operates some aspects of its SM. The organization under study conducts regional flights and has ground handling and maintenance capabilities, the depot maintenance excluded. During the current research, metrics were applied and accompanied with interviews in order to assess: the supportive or interfering role of safety management; management of resources allocated to safety investigations; commitment to realization of safety recommendations; timely communication of safety critical information to the lowest functions of the organization; sharing of common perspectives and views among SM and accident/incident investigators. The aforesaid list of metrics was selected based on the type of the information available in the safety investigation reports of the organization and the data recorded in regard to the investigation process. The literature review section below refers to SM topics that are relevant to the aforementioned aspects assessed, hence a discussion about all SM activities was out of the scope of this paper.
There might be a diversity of data recorded in Safety Management Systems (SMS) and included in safety investigation reports, and safety investigation procedures may vary across different organizations. The objective of the current research is to demonstrate an alternative way of exploiting available data from accident and incident investigation processes and reports in the context of an internal SM performance-based evaluation, and not to suggest an exhaustive list of metrics or an external benchmarking framework. Also, the findings of this study regard specifically the organization studied, therefore they cannot be generalized.

Safety management performance assessment
Performance evaluation enables organizations to measure whether their activities have the expected outcomes, and assess the necessity for implementing remedies. Interventions may be required in order to align a system with its initial design, embrace new technology and adapt to the physical, social, political and financial environments (e.g. Carton & Hofer, 2006;Parker, 2000). Following a comprehensive literature review, Kaspers et al. (2016) enlisted the quality criteria of metrics to be used for assessing performance based on a thorough theoretical framework; specific in what is measured; measurable; valid; immune to manipulation; manageable; reliable; sensitive to changes in conditions and cost-effective.
As an inextricable part of business functions, SMS are assessed mainly through quality assurance processes whose role is to evaluate SM and performance and drive respective improvements (BSI, 2007;ICAO, 2013a;ILO, 2001). However, as Tyler (2007) noticed, SMS-related information is sometimes hard to collect and difficult to transform into figures. Having recognized the need to move beyond compliance, the International Civil Aviation Organization (ICAO) suggested a transition to performance-based SMS assessments (ICAO, 2013b). Under the same concept, the European Aviation Safety Agency (EASA) communicated its intent to complement the SMS prescriptive regulatory framework with a performance-based environment (EASA, 2014).
Two examples of tools designed to support the transition to performance-based evaluations are the Safety Management System Evaluation Tool developed by the Safety Management International Collaboration Group (SMICG, 2012), and the Effectiveness of Safety Management (EoSM) instrument devised by Eurocontrol (2012). As a first step, these initiatives have successfully embodied the basic Plan-Do-Check-Act quality model, thus depicting the maturity of the system. However, they do not address the interdependencies of the SM activities and they approach SMS in a linear manner (Karanikas, 2016).

The role of senior and safety management
In all organizations, senior management is responsible for defining safety policies and procedures, allocating the resources required to accomplish safety activities, adapting best industry practices and incorporating regulations of state and international authorities (e.g. CASA, 2005;FAA, 2006;ICAO, 2013b;Goglia, Halford, & Stolzer, 2008;Ridley, 2008). The role of SM concerned, safety personnel should not interfere with operational decisions and remedial actions; since safety staffs do not directly control operational aspects, it is the principal duty of department managers to devise and implement solutions for safety deficiencies (Channing, 2008;Ferret & Hughes, 2011;Karanikas, 2014). Persons responsible for affected functional areas must be directly involved in the decision-making process and assigned with accountability for implementing appropriate corrective actions (CAA, 2002;Manuele, 2003;Stranks, 2008;TC, 2002). This way, functional directors get involved in safety processes and operationalize their safety responsibilities in their area.
Typically, the proper management's role is verified through documentation checks that focus on the clarity of organizational policies, and accountabilities and responsibilities of staff towards safety (e.g. CANSO, 2014;FOCA, 2013;TC, 2005). Also, safety culture and climate surveys are used to measure qualitative and intangible characteristics of SMS, such as management's commitment and communication of safety information (e.g. Arezes & Miguel, 2003;Ferret & Hughes, 2011). To date, literature and practice do not suggest methods for assessing whether SM exert its expected role as this can be reflected in the way recommendations after safety investigations are addressed.

Safety investigation resources
Accident and incident investigations comprise a fundamental SM practice and their contribution in safety assurance is highly valuable. The distinct role of safety investigations in SMS stems from their potential to uncover causal factors and present the aftermaths as derived from analysis of factual data. Manuele (2008) noted that high quality investigations, in terms of depth, clarity, punctuality and objectivity, along with management support in realizing remedial actions, affect decisively an organization's safety culture.
Resources allocated to investigations determine their extent and depth; as ICAO (2013b) recognized, available resources will curtail some safety investigations. However, current SMS performance assessment guidelines (e.g. CANSO, 2014;FOCA, 2013;TC, 2005) have not addressed methods for evaluating adequate resource allocation to safety investigations. Similar to the assessment of other SMS requirements, management's commitment to sufficient resource allocation is confirmed mainly through checks of relevant records.

Safety recommendations
Safety assurance activities include development and implementation of corrective actions in response to deficiencies that may affect safety (ICAO, 2013b); under this concept, the formulation of safety recommendations is the ultimate goal of accident and incident investigations. ICAO (2003) set specific requirements for safety recommendations: they must be addressed to the most proper operational or management level that holds the authority to proceed to the necessary changes; the suggestions must address objectives instead of specific actions in order to meet objectives; the recommendations must be developed following a dialogue amongst the involved parts in order to avoid unexpected and undesirable denial and resistance to their implementation.
Although SMS assessment guidance includes the verification of the implementation of corrective actions (e.g. CANSO, 2014;FOCA, 2013;TC, 2005), there has been no explicit reference to the measurement of recommendations' delivery time against predetermined and agreed deadlines. Thus, organizations are not assessed in regard to delays in recommendations' implementation, such delays possibly maximizing the exposure to risks identified through safety investigations.

Safety information communication
Safety communication is an inextricable part of a well-operated SM. TC (2004) coupled good communication and effective training with increased probability of a successful SM. Under this requirement, all organizational levels and functions must be aware of the strengths and weaknesses that affect operational activities. The Institute of Leadership and Management (ILM, 2003) heightened the necessity to guarantee continuous information flow to end-users as means to increase awareness of the hazards recorded and the corrective actions planned. Moreover, the information must not be restricted to safety topics; in a mature organizational culture, employees need to be knowledgeable about total organizational performance and benchmarking results (Karanikas, 2014). Inclusive electronic databases are expected to allow employees to retrieve information about industry and international standards, organizational plans and their incarnation progress, operational procedures, quality assurance findings and remedial actions (e.g. CANSO, 2014;FOCA, 2013;TC, 2005).
Specifically, in the context of safety investigations, authorities (e.g. OHSD, 2002), authors (e.g. Kletz, 2001) and professional practice (e.g. Brooker & Cooper, 2014) have recognised the value of disseminating relevant information to workers via structured reports and unstructured discussions. The goal is to maintain organizational memory alive, circulate aftermaths and increase risk awareness so that similar negative events can be avoided. Although safety communication is a crucial part of SM as a means to increase awareness about safety issues across the organization, measurements of safety communication timeliness have not been yet mentioned in the literature.

Scope and general approach
The present research was conducted at an aviation organization (AO) and explored how it could use data from safety investigation processes and reports in order to develop relevant SM performance metrics, in addition to the measurement of accident and incident rates and frequencies of causal factors. The AO's hierarchical structure includes: senior management, where the safety directorate resides; three middle management sectors, each supported by a safety department; air operations, maintenance, logistics and ground support units, each reporting to a section and running a safety office ( Figure 1).
The researcher exploited data from the AO's safety investigation progress records, investigation team reports submitted to the AO's safety directorate, final investigation reports released after the processing of investigation team reports and recommendation logs. The metrics employed to evaluate aspects of SM and the corresponding datasets analyzed are mentioned below in section 3.2. The significance level for the statistical tests was set to a¼.05. The results following the implementation of the metrics were accompanied by interviews with safety staff, as explained in section 3.3. (i.e. 42 accidents, 449 serious incidents and 319 incidents). All data were associated with aircraft accidents and incidents which occurred either during flight or on ground. The selection of the metrics described below was driven by the information included in the records, reports and logs provided by the AO, with reference to the literature suggestions regarding various aspects of safety management.
The overarching idea was the use of already available data from the organization's documents as a means to derive metrics that reflect the effectiveness of its safety management. Hence, the goal of the study was not to generate metrics that require recording of additional information during safety investigations and respective changes in the investigation process. Rather, this study exploited existing information not previously used by the organization on the scope of assessing aspects of its SM and indicating areas for improvement.

Metric 1: Duration of safety investigation phases
This metric regards the time elapsed amongst the several phases of accident and incident investigations. A considerable deviation from the foreseen deadlines could be attributed to mismanagement or lack of resources in the investigation process, or unrealistic expectations. According to the AO's safety investigation procedures: The investigation team shall submit its report in 50 days' time after the event's occurrence, accompanied by comments from the operating unit involved and/or affected by the accident or incident. The combination of the report and comments constitute the draft investigation folder. Afterwards, the sector which the operational unit reports to, must comment on the investigation folder in 20 days. This additional commentary also becomes part of the draft investigation folder. Next, the senior directorates addressed during the safety investigation are asked to add comments in the investigation folder in 20 days. Directorates' comments supplement the investigation folder too. After all commentary is collected, the safety directorate must publish the final investigation report in 60 days. Taking into account the time line referred earlier, along with an allowance of 20 days for secretarial procedures, the safety directorate must issue the official report in 170 days after the date the safety event occurred.
The sample included records from the progress of 475 out of the 810 accident and incident investigations due to lack of data for the whole sample. The dates between each of the investigation phases mentioned above were derived from the records, and calculations of medians where performed due to nonnormal distribution of the data as resulted from the Kolmogorov-Smirnov statistics.

Metric 2: Timeliness of final investigation reports' communication
The specific metric regards the time required for communicating the final investigation report to end-users at operating units and departments. The AO distributes the reports in hard copy format and imposes documentation controls in order to avoid publicity of the investigation reports and negative implications on persons and the organization as a whole. Albeit the AO has not set a specific timeframe for the communication of final investigation reports, according to the literature cited earlier such a metric can be considered as indicative of SMS performance.
For this metric, 89 records regarding an operating unit were analyzed. The unit was representative of the AO's operations in terms of extent and complexity of activities. The dates between the release of safety investigation reports and the communication of those at the operating departments were recorded and median values were calculated. The Kolmogorov-Smirnov tests revealed non-normality of data distribution.

Metric 3: Number and resemblance of recommendations
This metric regards two measurements: first, the difference between the number of recommendations stated in the investigation team reports and the ones included in the final reports; second, the number of common recommendations between investigation teams and the safety directorate. According to the AO's procedures, the recommendations generated by the investigation teams are not binding and are subjected to changes, additions etc. based on the comments received by the sectors and senior directorates and a final evaluation by the safety directorate. It is clarified that the AO provides safety investigation training to staff that has been already trained as safety officers and implement the risk assessment process of the organization as part of their duties. According to the safety investigation procedures of the AO, investigation teams are expected to formulate recommendations after evaluating various options, their possible effects on operability, side effects to other organizational functions, associated costs etc.
This particular metric would indicate the distance between the investigation teams and AO's safety directorate in terms of number and resemblance of recommendations. A significant distance could be attributed to flaws in information sharing amongst investigation teams and the safety directorate. This in turn, could imply ineffective communication across the organization.
Due to time limitations, the sample included 120 safety investigation reports out of the 810 reports provided by the AO. The reports were selected through systematic sampling with a sampling interval of 6 after ordering the original list of reports chronologically from the oldest to the newest. On the scope of this metric, two types of calculations were performed. First, the ratio of number of recommendations stated in each investigation team report per total recommendations found in the respective final report was calculated, followed by the computation of the median of ratios across the whole sample. Secondly, such ratios and a median value were calculated regarding the recommendations of the investigation teams that were adopted by the safety directorate and included in the final reports.

Metric 4: Types of recommendations
Taking into consideration that the literature proposes a supportive role of SM in developing remedial measures, each safety recommendation was classified as: Assignment: The recommendation states the objective to be achieved, meaning 'what' must be fixed. This type of recommendations indicates a supportive role of SM because the latter does not restrict managers in the way they will tackle the problems revealed during safety investigations. Action: The recommendation states specific methods to address a deficiency, thus minimizes the degree of managers' freedom to devise solutions. This indicates an interfering role of SM. Reminder: The recommendation refers to an existing rule/procedure which was not followed by the employees and its reinforcement is suggested. In this case, the role of SM can be perceived as supportive since it does not introduce an action (i.e. how the reinforcement will be achieved). Table 1 represents the above-mentioned classification accompanied with examples per type of recommendation. The frequency of each recommendation type would indicate to what extent the role of the AO's safety directorate has been supportive or interfering in operational managers' duties concerning the generation and implementation of corrective actions.

Metric 5: Timeliness of recommendations' implementation
This metric regards the time gap between delivery deadlines of recommendations and dates of their actual implementation, in total and per recommendation type (see Table 1). This metric would indicate potential delays in the implementation of corrective actions and trigger an exploration of underlying reasons.
For the calculations required for metrics 4 and 5, the recommendations included in the 120 reports analyzed for Metric 3 were used. The metric 4 concerned, frequencies of the recommendations types were calculated. Regarding Metric 5, 15 days for secretarial and administrative procedures were added when the final safety investigation report requested implementation of recommendations upon receipt. The Kruskal-Wallis tests were performed in order to explore any association between the type of recommendation (i.e. Metric 4) and the speed of its implementation (i.e. Metric 5).
The SM aspects assessed in this study and the respective evaluation methods discussed above are presented in Table 2.

Interviews
Following the analysis of the datasets, the research methodology and the results were communicated to seven safety professionals of the AO as follows: the chief of the safety directorate, the head of the safety investigations office, a safety officer positioned at a middle management sector, two safety officers working at an operating unit and two safety investigators who were not holding any SM-related role. Afterwards, individual interviews were scheduled in order to discuss the results and welcome the interviewees to explain the findings. The interviews were unstructured, were held at places and times convenient for the subjects, lasted between 50 and 60 min and the researcher kept notes which were later verified by the participants. The interviewees did not give their consent for visual or audio recordings of the discussions.
The researcher aimed at combining perspectives from safety professionals with various roles in the AO; comparison of views and evaluation of potential differences amongst the interviewees were out of the scope of the study. All the aforementioned persons were experienced accident and incident investigators Table 2. Metrics, aspects and methods for safety management evaluation.

Metric SM aspect assessed Evaluation method
Duration of safety investigation phases Commitment of resources to safety investigations.

Applicability of safety investigation procedures
Deviations of the duration of investigations from the one foreseen in the organization's documentation.

Timeliness of final investigation reports' communication
Timely communication of safety critical information.
Time required to distribute the investigation reports to the lowest functions of the organization. Number and resemblance of recommendations Extent of common perspectives and views among SM and accident/incident investigators.
Number and similarity of published safety recommendations to the ones suggested by the investigation teams. Type of recommendations Role of SM (i.e. supportive or interfering). Frequency of respective recommendation types stated in safety investigation reports.

Timeliness of recommendations' implementation
Commitment to realization of safety recommendations.
Differences between due and implementation dates of recommendations. The SM directorate assigns to a planning or operating function the responsibility for developing a corrective action as means to resolve an identified flaw.
The logistics department must resolve the problem of ineffective personal protective equipment (PPE) against noise.

Supportive
Action (a specific solution is stated) The SM directorate develops a corrective action and requires its implementation.
The logistics department must replace the current noise reduction PPE.

Interfering
Reminder (a reinforcement of rule/procedure is stated) The SM directorate refers to an already established procedure or rule and suggests its reinforcement.
Line managers must stress to their subordinate employees the requirement to use PPE.
Supportive and four of them were holding SM-related positions at the time of the interviews. This way the author ensured diversity and representativeness of standpoints.

Results
The results from the analysis of the data and the corresponding comments of the interviewees on the findings are presented in the following per metric.

Metric 1: Duration of safety investigation phases
The search for potential delays in the safety investigation stages resulted in the figures shown in Table 3. It is clarified that although the data were not normally distributed, a short distance was observed between the median and mean values compared to the dispersion of the data. This indicates that the findings were representative of the whole sample.
The results showed that the organization under study had experienced severe delays in its investigation phases at the operating unit and middle management sectors. All interviewees attributed those findings to ineffective resource management at the aforementioned organizational levels. Particularly, although investigation team members should be released from their normal duties during each safety investigation, this was not practiced by the managers of the operating units. At the middle management sector, the delays were linked to understaffing and the requirement for the accomplishment of a variety of activities in addition to the coordination of the commentary of safety investigation folders. Safety staff of the operating unit and safety investigators claimed that the safety investigation procedures applied across the whole AO is not scalable and flexible enough to account for the variety of special conditions in each section and operating unit.

Metric 2: Timeliness of final investigation reports' communication
In average, each report was communicated to the end-users of operating units in 11 days; communication of final safety investigation reports to the end-users did not show important delays, taking into account secretarial procedures. The AO's safety personnel stated that the organization recognizes the merit of effective and timely communication of investigation reports across all organizational levels as a means to prevent unwanted events through the aftermaths formulated in such reports.

Metric 3: Number and resemblance of recommendations
The final investigation reports included 48% more recommendations than the ones investigation teams formulated. 61% of the recommendations proposed by the investigation teams were stated in the final reports. During discussions on this topic, AO's safety staff pointed out that safety investigators put much effort in their tasks and are highly concerned about the quality and completeness of their reports. However, investigators do not acquire the 'big' picture of the organization, in terms of complexity and resource constraints. Moreover, investigators are not able to estimate costs when they design recommendations and they are not aware of any other planned corrective actions that possibly overlap with the remedies proposed by the teams.
The interviewees further attributed the findings in the incomplete information the investigators obtained regarding the organization's plans, initiatives, constraints etc. This in turn was ascribed to the lack of a central data storage system where such information could be stored and retrieved. Additionally, the safety directorate had not communicated to the investigators the reasons of the differences between what the investigation teams suggested and what management adopted, because the AO lacks relevant procedures.
Safety staff of the safety directorate claimed that although the AO's procedures describe the distinct roles of several functions in the safety investigation process and generation of recommendations, results from corrective actions' monitoring had shown that those roles had not been practiced. Operating units and/or middle management sectors had delayed, or even unilaterally cancelled, corrective actions without providing relevant feedback to the safety directorate. Consequently, the safety directorate had been concerned that the deficiencies revealed through investigations would not be timely or at all addressed, and subsequently, in many cases the specific directorate undertook the role of managers.
The rest of the interviewees acknowledged that safety recommendations were frequently strict and did not allow flexibility to operating units and middle management levels in the implementation of remedies. These participants added that sometimes the 'Action' recommendations were not matching the special conditions, resources and other factors of the various operating units, thus increasing occasionally the implementation time and possibly the quality of the corrective actions. The frequency of 'Reminder' recommendation types in final investigation reports was perceived by the AO's safety personnel as positive. They claimed that it was not necessary to overwhelm other organizational functions with publishing additional directives regarding reinforcement of established procedures and rules.

Metric 5: Timeliness of recommendations' implementation
Managers implemented recommendations 1 month after the publication of the final safety investigation reports. The recommendations' delivery deadline defined in those reports had a median value of zero (0). The Kruskal-Wallis test showed that 'Assignment' type recommendations needed more time for implementation, followed by the 'Action' and 'Reminder' type ones (v 2 ¼10.600, dF ¼ 2, p¼.005). The same order was calculated by the Kruskal-Wallis test for the time allotted by the safety directorate for the realization of each recommendation type (v 2 ¼90.597, dF ¼ 2, p¼.000).
The AO's safety staff anticipated the aforementioned results, which suggest that safety directorate requested almost immediate implementation of recommendations. The interlocutors argued that most of the 'Action' type recommendations regarded easy to implement changes (e.g. subtle amendment of procedures) and 'Reminder' type measures required by default short implementation time. Since 'Action' and 'Reminder' recommendations held 78% of the total number of the recommendations published by the AO, the short average implementation time revealed by the metric was expected. On the other hand, 'Assignment' recommendations had usually referred to introduction of new technology or technical modifications, extensive changes of procedures and further research for deficiencies identified during safety investigations. Such recommendations required detailed planning and research, and consequently increased time for their delivery. However, they held about one fifth of all recommendations and they did not significantly affect the results of the specific metric.

Discussion
The analysis of data from safety investigation reports and processes in combination with the discussions held with the organization's staff revealed both positive and negative performance of the SM aspects considered in the study. The significant delays in safety investigations were credited by the AO staff to ineffective resource management and investigation procedures, which had led to a gap between AO expectations and actual deliverables. Although the literature suggests that timely and adequate allocation of resources will benefit organizations in terms of depth and speed of accident and incident investigations (see section 2.3 above), it seems that the specific organization had not realized the extent to which such resources were not always available or committed to investigations.
Hence, even though the intentions of the AO to derive lessons from safety investigations in a timely manner were aligned with the views expressed in the literature, this was not usually feasible. Perhaps, taking into account that the investigation of accidents and serious incidents are mandatory in the aviation sector, the investigation of fewer incidents and the allocation of more recourses to the investigation of safety events of higher severities could have allowed the organization to achieve its objectives. However, the aforementioned practice could have deprived the AO from aftermaths obtained from incident investigations; thus, a balance between expected benefits and resource investment should be considered. The latter could be reflected in the procedures of the AO by allowing different investigation timelines per severity class.
The fact that 48% more recommendations were stated in the final reports compared to the number of remedies stated in the investigation team reports, and that only 61% of the latter were adopted, indicated a dissociation amongst the safety directorate and investigators. Although the AO expects from investigators to be aware of the wider organizational context when they formulate recommendations, the quantitative and qualitative differences in the generation of safety recommendations were attributed to the lack of consistent information sharing between senior/middle management and investigators.
The findings related to the number and resemblance of safety recommendations reflect the literature references which claim that communication of inclusive information across all organizational functions is of high importance; such information should not to be restricted to safety topics (see section 2.5 above). The lack of a central information system did not support investigators' awareness of the overall organizational context and led to the proposal of remedies which were not completely aligned with the plans, constraints and other conditions of the AO. In addition, it seems that, even under the lack of such a central system, a bidirectional communication between the safety directorate and investigators could have alleviated over time the discrepancy in regard to the quantity and quality of safety recommendations. Thus, the organization had missed the opportunity to minimize the aforementioned gap over time.
On the positive side, the quick dissemination of safety investigation reports to the end-user level and the timely implementation of safety recommendations were attributed correspondingly to the appreciation of communication of such information across the organization, and the importance given to the efforts for preventing future accidents and incidents. It seems that the AO had successfully estimated the time planned and the resources allocated for the implementation of remedies. It is noticed that the corresponding metric employed in this study (i.e. timely implementation of safety recommendations) does not account for the quality and effectiveness of the remedial actions, which was not possible to be evaluated through the analysis of investigation reports and records. Nonetheless, the significance of the aforementioned dimensions is pointed out in the literature (see section 2.4 above).
The relatively high percentage of 'Action' type recommendations indicates that the AO's safety directorate had played an interfering role in the responsibilities of other departments, in contrast with the literature suggestions (see sections 2.2 and 2.5 above). Such an interfering role of the safety directorate was the result of inadequate commitment of managers to the realization of 'Assignments' type recommendations in the past. This had resulted to important delays in the implementation of remedies and increasingly forced the safety directorate to formulate safety recommendations in a way that those dictated what should be performed instead of stating what should be achieved. Such an approach literally violated the scope of the recommendations as referred in standards and literature; the AO's staff attributed the aforesaid evolving practice of the safety directorate to the lack of a productive dialogue across the various organizational levels.
Since similar research across the aviation or other sectors was not found, it was not possible to evaluate whether the aforementioned findings reflect normal practice in the industry. The metrics suggested in this study satisfy most of the quality criteria presented in the literature review (see section 2.1 above). The metrics employed in this research are Grounded on a theoretical framework which combines literature and standards Specific in what is measured, as explained in Tables 1 & 2 Measurable, as it was demonstrated in the methodology and results Valid since the results were accepted and confirmed by the interviewees Immune to manipulation because they are based on documented data Reliable since they depend on existing records and do not require interpretation of data; the classification adopted in Table 1 was deemed by the interviewees clear and is not expected to confuse the analyst. Reliability can be affected by the quality of data (e.g. mistyping, missing data), but the analysis of a large sample is expected to compensate for such problems Manageable because they did not require the maintenance of additional data from the organization and they employ simple calculations Cost-effective; the time the researcher invested on the collection and analysis of data equals to 20 working days, which, when considering the sample size and the significance of the findings, was perceived by the safety staff as reasonable. However, the cost-effectiveness of the metrics presented in this paper depends on the amount of records to be processed and the extent of the surveys to be performed in order to explain the results It is noted that the sensitivity to changes in conditions, which is the last quality criterion, could not be assessed in the frame of this study. This can be evaluated if the metrics are applied periodically or after reformations in SM practices.

Conclusions
This research demonstrated how organizations might use data from safety investigation records and reports in order to develop metrics for assessing performance of various SM aspects in addition to event rates and frequencies of causal factors. The findings from the analysis of such data maintained by the aviation organization under study triggered respective discussions, through which positive and negative areas of SM performance were identified. Lack of safety ownership, inadequate communication amongst organizational levels, ineffective resource management and non-scalable procedures were the main flaws recognized by the safety staff after they were informed about the numerical results of the metrics. Although the aforesaid problems were known beforehand, the extent to which they had affected various aspects of SM was not obvious before the implementation of the metrics proposed in this paper.
The metrics that were applied in this research have not been previously suggested in literature and practice, contribute to a performance-based approach of SM evaluation, satisfy most of the quality criteria, as presented in the discussion section, and are based on data an organization might maintain but not exploit. Such metrics might comprise a basis for the development of indicators that will enable organizations to monitor various activities of their SM periodically or between milestones. The set of metrics, frequency of their monitoring and degree of engagement of organizational functions in the interpretation of results are dependable on the available resources and culture.
It is clarified that the findings of this study cannot be generalized to the aviation industry since the metrics were applied to a single organization. Also, each organization might record different data in regard to safety investigations, so the implementation of the whole set of metrics presented in this study might not be always feasible. However, this paper presented a method that organizations, regardless of industry sector, can follow in order to develop metrics depending on the data they maintain in relation to safety investigation reports and processes, and use those as a means to improve their safety management. Quality of safety recommendations and depth of investigations are examples of aspects that can be also evaluated depending on the resources and type of data available. Nonetheless, it is of paramount importance the results of such metrics to be followed by interviews and/or questionnaire surveys in order to interpret figures and inform decisions. Raw numbers without contextual information will not allow organizations to capitalize the investment required to develop and monitor any type of metric.
The author does not suggest ceasing the analysis of safety investigation reports in order to classify causal and contributing factors and calculate accident and incident rates. However, the aforementioned practices focus principally on safety outcomes and do not offer the opportunity to assess SM performance in the way that this paper demonstrated. The approach of this study can be linked to data-mining methods through which companies, especially in the aviation sector, distil useful information from the analysis and combination of various data sources, and monitor those as a means to improve safety and efficiency (e.g. Marthur, 2002;Nazeri, Bloedorn, & Ostwald, 2001;Pagels, 2015).

Disclosure statement
The author reports no conflicts of interest. The author alone is responsible for the content and writing of this article.