Statistical methods – still ignored? The testimony of Swedish alumni

Researchers have promoted statistical improvement methods as essential for product and process improvement for decades. However, studies show that their use has been moderate at best. This study aims to assess the use of statistical process control (SPC), process capability analysis, and design of experiments (DoE) over time. The study also highlights important barriers for the wider use of these methods in Sweden as a follow-up study of a similar Swedish study performed in 2005 and of two Basque-based studies performed in 2009 and 2010. While the survey includes open-ended questions, the results are mainly descriptive and confirm results of previous studies. This study shows that the use of the methods has become more frequent compared to the 2005 study. Larger organisations (>250 employees) use the methods more frequently than smaller organisations, and the methods are more widely utilised in the industry than in the service sector. SPC is the most commonly used of the three methods while DoE is least used. Finally, the greatest barriers to increasing the use of statistical methods were: insufficient resources regarding time and money, low commitment of middle and senior managers, inadequate statistical knowledge, and lack of methods to guide the user through experimentations.


Introduction
Effective statistical methods for improving the quality of processes and products have been around a long time (Box, Hunter, & Hunter, 2005). Statistical process control (SPC), process capability analysis (CA), and design of experiments (DoE) are powerful problem-solving methods available in the quality engineering toolbox. SPC (Shewhart, 1931) and DoE (Fisher, 1925) have been around for almost a century while CA has a history of at least forty years (Juran, 1974). Well-known textbooks describe all three methods thoroughly. Montgomery (2013b) for example, cover SPC and CA, while, for example, Montgomery (2013a) and Box et al. (2005) cover DoE. We use the following definitions of the three methods; (we treat affiliated sub-methods or tools as secondary aspects of the main methods to facilitate understanding): specifications. Typical tools within CA are process capability indices such as C p and C pk . . The purpose of DoE is to plan, conduct and analyse experiments systematically and statistically to improve products and processes. Frequent tools in DoE include factorial designs, especially two-level factorial designs, response surface methodology, and Conjoint Analysis.
SPC and CA are methods that are more directed toward monitoring existing processes and assessing their capability, while DoE typically is used in product and process improvement and innovation. These methods have traditionally been used most extensively in manufacturing, but they are also used elsewhere, for example in the service sector (Gustafsson, Ekdahl, & Bergman, 1999). While each of the three methods is useful by their own, practitioners can cyclically combine them to create a powerful problem-solving process as shown in Figure 1. Hence, this study focuses particularly on the use of these three statistical methods.
Statistical methods are essential for monitoring and improvement, but can only create value if used. However, even though the literature on the use of statistical methods show mixed results, we conclude that practitioners use SPC, CA, and DoE sparingly at best. The low use frequency is particularly the case if we define use as a systematic application of the methods. All identified literature covering quantitative investigations of the use of SPC, CA, and DoE are conducted in Europe, and mostly within Sweden. In a survey of Swedish industry, Deleryd (1996) found that 47% of the responding companies had used CA. Gremyr, Arvidsson, and Johansson (2003) interviewed product development managers and quality managers from 105 Swedish manufacturing companies. Gremyr et al. (2003) found that 53% of whom stated that their firms used DoE and 60% of whom reported the use of SPC and CA. Bergquist and Albing (2006) studied the use of SPC, CA, and DoE in Swedish companies that employed alumni of Luleå University of Technology (LTU). In that study, 44% of the participating companies had tested SPC while 33% and 23% had tested CA and DoE, respectively. However, when defining use as something more than just occasional trials, these numbers fell to 27% for SPC, 21% for CA, and 17% for DoE (ibid). Bergquist and Albing (2006) also found that a lack of knowledge and competence among engineers was a major barrier to increased use of statistical methods. In a later study targeting Swedish production managers, Poksinska, Pettersen, Elg, Eklund, and Witell (2010) found that the studied organisations seldom used SPC

Design of Experiments (DoE)
Process Capability Analysis (CA)

Statistical Process Control (SPC)
Can the process capability be improved?
How does the process perform compared to product specifications?
Can the improved process be monitored? and DoE and that many respondents did not recognise these methods. A potential issue with the Poksinska et al. (2010) study is that around a third of the responding production managers were unfamiliar with the statistical methods, and may not have been suitable respondents.
In a study of Basque industries in Spain, Tanco, Viles, Alvarez, and Ilzarbe (2008) found that while 94% of the participating organisations conducted experiments, only 20% used statistically based experimental designs. The same researchers later investigated the gap between the rapid theoretical development of DoE methods and their slow uptake in commercial practice, see Ilzarbe (2010, 2009). The Basque studies identified 16 barriers that hinder the industrial application of DoE. Two of the most important barriers were a low commitment among managers and the users' limited statistical knowledge.
As a complement and augmentation to the identified studies, the purpose of this article is to analyse the use and related barriers to using SPC, CA, and DoE. The main contribution of this study is an added time-perspective, by a comparison with previous Swedish studies, such as Bergquist and Albing (2006). Another major contribution is to connect the usage results with a list of barriers assessed by the Swedish respondents and to compare the results with those found by Tanco et al. (2010Tanco et al. ( , 2009.

Method
We distributed a survey that targeted alumni of the Master's programme in Industrial and Management Engineering at LTU who graduated between 2005 and 2013. We chose this population based on four criteria. First, the programme features a compulsory course in quality management with SPC, CA and DoE in their syllabus. In addition, many of the students also take an optional advanced course on SPC, CA and Six Sigma methodology as well as a course in DoE. All the respondents should thus have at least a basic understanding of the methods and are therefore likely to understand the survey questions. The respondents should also be able to determine whether their organisations use the methods and give a fair assessment of the methods' applicability in their organisational contexts. Second, the respondent choice is comparable with that of Bergquist and Albing (2006), and a longitudinal perspective on the methods' use can be obtained. Note, however, that the participants in this study represent a more recent group of alumni. Third, we expected the alumni from our university to be more willing to participate in the study, yielding a higher response rate. Last, the selected population was convenient to sample because the university maintains a database of alumni.
The questionnaire was split into four different sections containing questions on the respondent's background, their two most recent workplace(s)e; their prior use and knowledge of SPC, CA, and DoE; and barriers they considered to hinder the use of statistical methods. Before we sent the questionnaire to the respondents, we let to four former students of the LTU programme and four researchers/teachers review it for feedback and critical assessment, and then revised the questionnaire. Each respondent got a copy of the revised questionnaire by mail together with a cover letter and a stamped return envelope. We asked them to return the completed paper questionnaire or to complete a web-based version.
In the questionnaire, we requested the respondents to quantify the methods' use in their last two workplaces using Likert scales ranging from 1 to 5. The number 1 indicated that the method was 'not used at all' and 5 designated that it was 'used systematically in all relevant processes.' The respondents could also answer 'I do not know' to each question. After having assessed the methods' use, we asked the respondents to evaluate the relevance of each method in their organisation's work.

Total Quality Management 247
The questionnaire also included questions about the respondent's knowledge and use of SPC, CA, and DoE. As before, the respondents had to quantify their responses using a Likert scale ranging from 1 to 5. The answer 1 meant the respondents had 'no knowledge' of the statistical method in question or that they had 'never used the particular statistical method.' The answer 5 meant that they had a 'very good knowledge' of the method or had 'used the method in question more than ten times.' Answers 2 to 4 thus meant flawed to good method knowledge. The appendix provides an excerpt from the questionnaire illustrating English translations of questions related to the prior use and knowledge of methods, assessed utilisation in organisations, and barriers.
In November 2013, we sent the questionnaire to 410 respondents, all of whom worked in Sweden. The cover letters were personalised by writing the respondent's name at the top to increase the response rate. We gave the respondents two weeks to answer the questionnaire and got 107 responses. The reminder sent out in December 2013 yielded 40 additional responses. In total, we obtained 147 responses, giving a response rate of 36%. We received 91 completed paper questionnaires, and 56 used the web-based option. The 147 respondents assessed the use of the methods in 211 workplaces within 136 different organisations.

The respondent population
We classified the respondents' workplaces according to branch, size (medium or large), and type of organisation (industry or service) based on the SNI 2007 classification system (Statistics Sweden, 2007), see Table 1. Of the 211 workplaces, 26% were medium sized (i.e. had fewer than 250 employees), while 74% were large. Further, the SNI codes classified 67% of the workplaces as industrial and 33% of the workplaces as service organisations. The respondents consisted of 65 women and 146 men aged between 24 and 41, with an average age of 31. We note that our sample of workplaces has a clear bias toward larger organisations compared with the regular Swedish organisational structure. For example, of all registered companies in Sweden, 99.9% of all companies are between 0 and 249 employees (Statistics Sweden, 2007). However, about 57% of all employees in Swedish companies work in organisations with more than 200 employees, and 27.9% of the Swedish work force is employed by the public sector (Statistics Sweden, 2007). The sample, therefore, does not deviate more than what can be expected, given that the respondents were engineering programme alumni. Differences, for instance, include that 76% of respondents work for organisations with more than 250 employees, and manufacturing organisations are more common in our sample compared with the Swedish business population. Tables 2 and 3 present the respondents' self-assessed levels of statistical method knowledge and their self-assessment of their use of the methods. Fifty percent or more of the respondents considered themselves to have at least basic knowledge of the methods, with SPC appearing to be most widely understood. Some respondents considered themselves to lack knowledge of the methods despite that they were introduced to them during their university studies.
Furthermore, between 25% (DoE least common) and 52% (SPC most common) stated that they had used the methods actively more than once. We conclude that respondents on average have a reasonable theoretical knowledge of the methods, but they have less practical experience of using the methods, especially in the DoE case. Figure 2 presents the reported use of the methods. We first consider 'use' to mean that the method had at least been tested occasionally and responses stating ' … use cannot be  Large organisations (i.e. those with more than 250 employees) use statistical methods more often than smaller organisations. Figure 3 shows that the use of statistical methods is more frequent in the industry than in the service sector. The difference is especially obvious for SPC, which 81 of 102 industrial workplaces (79%) use. While we rarely found service sector organisations using statistical methods, they were not absent. Figure  3 suggests that industrial organisations use CA more systematically than they use SPC and DoE, possibly because quality management systems standards such as ISO/TS 16949 require analyses of process capability.   Respondents who considered their method knowledge flawed or non-existent report lower use. Their lower reported use may, of course, relate to difficulty in recognising the use of unfamiliar methods in the organisation. The flawed knowledge may also be because this sub-group of respondents could be more prone to work in organisations where these methods are uncommon. We, therefore, examined the reported use rates for the subsample of respondents who reported at least a basic level of knowledge of each method when we first had excluded the respondents stating ' … use cannot be assessed.' Within this sub-sample, the reported use of the methods increased to 58% for SPC, 47% for CA, and 35% for DoE. In conclusion, the overall reported use of statistical methods increases by between 2% and 6% if we only consider responses from respondents reporting at least a basic level of knowledge.   Figure 5 presents the respondents reported the use of the methods for workplaces where they think that the methods would be relevant. Around 50% or more of the respondents said that they considered statistical methods relevant to their organisations. We here define the 'potential' for increasing the usage of the methods as the percentage of workplaces where the methods were considered relevant but were not used or had only been occasionally tested. Based on this definition, DoE has the highest potential (17%), followed by SPC (16%), and CA (10%). Expectedly, the use and workplaces where the respondents consider the methods relevant are also positively correlated.

Longitudinal comparison
To compare our results with the findings by Bergquist and Albing (2006), we define the 'use of methods' in the Bergquist and Albing (2006) study as a method that is at least ' … used, but only occasionally in a few processes' or more frequently. The usage proportions reported in the Bergquist and Albing (2006) and this study are shown in Table 4, together with p-values from the statistical test. We can create a statistical test with the null hypothesis that the two proportions are equal under the assumption that the sampling properties are equal, and calculate a confidence band for the currently observed proportion. The calculations for the statistical test were performed in the Minitab 17® package using Fisher's exact test and a 5% significance level.
The low p-values for SPC and DoE mean that we can reject the null hypothesis and that the use of SPC and DoE reported here is significantly greater than what Bergquist and Albing (2006) found. However, the higher p-value for CA means that we cannot reject the null hypotheses that the use of CA was unchanged.

Barriers to wider use of the methods
We also asked the respondents to rate how important they felt that 19 potential barriers were to widen the use of statistical methods. The respondents had to rate the importance of each barrier using a ten-point Likert scale with a higher score indicating greater importance. The 16 barriers of Tanco et al. (2010) were expanded to 19, partly because Tanco et al. (2010) only focused on DoE and partly because we considered some of their barriers to be excessively broadly defined. For example, we split the 'lack of resources' barrier into 'lack of time,' 'lack of money,' and 'lack of material.' The 19 barriers that the respondents were asked to rate are listed in Table 5, sorted according to their ranking in the work of Tanco et al. (2010).
As mentioned above, we asked the respondents to rate each barrier using a scale from 1 (no impact at all) to 10 (very substantial impact). Figure 6 presents the rates of each of the 19 barriers along with estimated 95% confidence intervals for the mean. The means of the 19 barriers (with 95% decision limits) show that the barriers with means significantly higher than the grand average (significance level 5%) were: B8 'insufficient time'; B1 'low commitment of senior managers … '; B2 'low commitment of first-line managers … '; B9 'insufficient monetary resources'; B5 'insufficient statistical knowledge … ' and; B6 'lack of methods to guide the user … .' Barriers with means significantly lower than the grand average (significance level 5%) were: B17 'statistics has a bad image … '; B3: 'poor education … to engineers'; B19 'prior negative experiences … ' and; B4 'the engineers" resistance to change … .' Moreover, non-overlapping confidence intervals in Figure 6 indicate that the corresponding barriers are significantly different on at least 5% significance level. For example, B8 'insufficient time' is significantly different from all barriers except B1 'low commitment of senior managers … .' Furthermore, a difference between this study and that of Tanco et al. (2010) was that B3: 'poor education to engineers' was the second least important barrier in our study but the second most important barrier increased DoE use in the Tanco et al. (2010) study. While the difference may reflect differences in the educational systems of Sweden and Spain, a likely reason is that our respondents were engineers who had taken university courses with the methods in their syllabus. Table 5. The 19 barriers, sorted in descending order according to their rankings in the study of Tanco et al. (2010), which are shown in parentheses and ranged from 1 (most important) to 16 (least important). Note that some barriers have the same ranking because they were formed by subdividing one of the original barriers in Tanco et al. (2010).

Barrier
Statement in survey B1 (T1) The low commitment of senior management due to lack of knowledge of statistical tools for process and product improvement B2 (T1) The low commitment of first-line managers due to lack of knowledge of statistical tools for process and product improvement B3 (

Answers to open-ended questions Benefits of using statistical methods
The respondents could also identify what they considered to be the most significant advantages of statistical methods. Such suggestions were given by 75% of the respondents. Around 44% of the respondents' answers could be summarised by saying that the methods facilitated fact-based decision-making. Some respondents further specified this statement by expressing that the methods made it possible to identify causal relationships (4%), led to better process understanding (11%), facilitated process monitoring and control (5%) or gave a wide-ranging overview of a data-rich environment (4%). The comment quoted below is representative: 'Statistical methods allow us to base decisions on facts and act before it's too late (with the help of trends, control limits, etc.)' Another frequently reported benefit was that respondents saw the methods as essential for process improvement (15%). Besides improved process knowledge (11%), many respondents saw a general understanding of variation as another benefit (11%). Many also mentioned the possibility to observe and predict changes over time (4% each) as benefits. A few also stated that one advantage of using the methods was that their statistical skills were maintained or improved.

Additional barriers
Besides the 19 rated barriers, we also asked the respondents to exemplify other barriers that they consider hindering the use of the statistical methods. Such additional barriers were provided by 27% of the respondents. We interpreted many of the answers as reformulations of barriers that had previously been identified, such as managers' and decision makers' lack of understanding of statistical theory, and disputes about the interpretation of the results. One respondent wrote: The major reason I think, is that too many senior managers lack university education; they have little knowledge of statistical tools and those who are aware of them are generally skeptical.
[The use of statistical tools] is not really part of their world. Comments like these, related to managers' lack of understanding of statistics were common. Other frequent comments relate to a more general lack of understanding of statistics, such as 'Knowledge is required to implement statistical methods, which is missing in many organizations.' Respondents also reported obstacles relating to organisational culture and resistance to change by individual employees, sometimes in ways that we link to barriers B4, B12 or B17: 'Culture! People often say there's no point and that what we do is way too complicated.' Other obstacles are harder to fit into the original barrier list, such as process specific obstacles or the perception among people employed in service organisations that statistical methods are not useful in their work. Some respondents also noted difficulties in acquiring data of sufficiently high quality or access to data in general. Lack of high quality, automatically sampled data means that any application of statistical methods would require manual sampling efforts beyond their company's regular measurement schemes. Another mentioned barrier was related to specific problems caused by the size of the industrial process, for example: 'Our processes are too large for Design of Experiments; a scrapped run would cost too much.'

Discussion
This article is mainly descriptive and should be considered as a follow-up of previous research. To further assess the items of the survey we performed a cluster analysis of the Likert-scale variables. Figure 7 show how they group into three classes. The assessments of the employer's use of the methods are correlated amongst each other, and so are the method knowledge of respondents and respondents' use.
If we assume that the items in some sense measure the same constructs, that is, statistical process improvement methods, it is possible to perform a test for internal reliability, which was done using the IBM© SPSS© software package Version 24. The inter-item correlation matrix is shown in Table 6.
The Cronbach's alpha for this inter-item internal consistency 'reliability' is 0.884, which should be considered high. Furthermore, performing a principal component analysis of the items in Table 6 using a cut-off of eigenvalues larger than one would retain only two principal components. The first component has an eigenvalue above five, so there is a high correlation among the variables. Respondents that are competent in one method tend to be competent in all, and the same goes for usage by both respondents and the companies. There is also a correlation, albeit weaker, between workplaces use, respondent use and respondent knowledge. The causality is, of course, hard to evaluate, i.e. if knowledgeable respondents seek workplaces where they can use their skills or if the skills are improved as they get involved with the methods when coming to such workplaces where they are used.
Our findings confirm previous results reported in the literature but also gives some interesting differences. It is therefore relevant to discuss any changes seen between our results and the reported tool use by Bergquist and Albing (2006) and the barriers reported by Tanco et al. (2010). Starting with the latter, Tanco et al. (2010) found 'lack of commitment among managers' and 'poor statistical background of the engineers' to be the most important DoE usage barriers. The comparatively modest impact of the engineers' statistical knowledge in our follow-up study might be due to the population that we surveyed, which consisted of graduates who had completed courses on statistical methods. Hence, compared with the Tanco et al. (2010) study the results from our study may be viewed as the testimony from the newly trained engineer's experiences entering the industry with theoretical knowledge about the methods he or she would like to apply. Indeed, our study also highlights the low commitment of senior management and first-line managers as an important barrier. However, the most important barrier in our study was insufficient time. Respondents also considered Insufficient monetary resources to be a problem.
The least important barriers in our study were 'bad image of statistics,' 'statistical tools are taught badly,' 'prior negative experiences with statistical tools,' and 'the engineers" resistance to change.' We argue that a possible explanation is that we survey engineers that received their training recently. These engineers may still be more open to new ideas, and may not have had many negative experiences of using the tools. The two least important barriers identified by Tanco et al. (2010) were 'not enough software aid' and 'previous bad experiences with DoE.' Again, we believe that the two surveys' potentially different study populations may explain some of the differences in the ratings. We conclude that the differing barrier rating results show that the same barriers do not apply to all European countries and for all respondents. A general barrier noted in both studies, however, is a low commitment of senior management and first-line managers.
Moreover, we saw that the use of statistical methods (especially DoE) differed between workers reporting a basic knowledge of statistical knowledge and those claiming a deeper understanding. The survey does not answer the question of whether competence prompts individuals to use these methods or if it is the other way around. The methods also have slightly different emphases, with SPC and CA being more production oriented while DoE focuses on improvement and (product) development. Since we asked the respondents if their organisations used statistical tools in all relevant processes in a similar manner, it is possible that the respondents thought about production processes where CA and SPC are relevant. The process or product improvement processes are more suitable for DoE, and the respondents may have had the production in mind when classifying the DoE relevance. If so, the use of DoE could be greater than what we found here.  We recognise that the LTU alumni who participated in this study can be argued to have a greater understanding of statistical methods than the average Swedish engineer. As noted by Bergquist and Albing (2006), the choice of respondents may induce response bias in the sense that respondents with formal training in statistical methods may be more prone to seek work in organisations interested in these skills. Such as bias would imply that we would probably have obtained a lower frequency of the methods' use if we had examined a random sample of Swedish engineers. However, we argue that the respondents must have at least some knowledge of statistical methods in the survey to avoid excessive researcher guesswork.
Other surveys measuring the use of statistical methods have achieved response rates between 16% and 83% (Bergquist & Albing, 2006;Deleryd, 1996;Gremyr et al., 2003;Poksinska et al., 2010). According to Baruch and Holtom (2008), the average response rate for studies that use data collected from organisations is 35.7%, (with a standard deviation of 18.8%). Since the response rate of this study (36%) lies within this interval, we consider the response rate as acceptable. We used Fisher's exact test to compare the answers we received before and after the reminder was issued. The difference between the answers the first 107 responses and the 40 that were obtained after the respondents were prompted a second time was not statistically significant at the 5% significance level, see Table 7. The reminder difference for the CA results are near the threshold, and all proportions were smaller when studying respondents obtained after the reminder. Although we cannot reject the null hypotheses that the proportions were the same, there is, of course, a chance that the respondents that feel that their organisations are using the tools are more eager to answer. In that case, the use may be smaller than obtained here. In the extreme case, that all non-responders were to answer that the tools were not used if prompted further, the difference between this study and the Bergquist and Albing (2006) study would disappear. We do, however, hold that to be highly unlikely.
We did not see any response differences through controlling for the background variables, such as gender and age. Finally, we once more want to stress that since the respondent population has a bias toward larger organisations, especially within industry, compared with the overall Swedish organisational structure. We mainly consider the tool use assessment and the identified barriers to reflect the situation in larger manufacturing organisations in Sweden and the results from our study show that larger organisations use statistical methods more frequently than smaller ones.

Conclusions
Our results show that the respondents report that 54% of their recent workplaces use SPC. The corresponding percentage for CA and DoE were 41% and 33%, respectively. If we only consider respondents claiming at least a basic understanding of the methods, the reported usage rises to 59% (SPC), 47% (CA), and 35% (DoE), respectively. These values are greater than those obtained in earlier studies on Swedish workers by Deleryd (1996), Gremyr et al. (2003). The values are also significantly higher for SPC and DoE compared with the study by Bergquist and Albing (2006). The increased use here suggests that the use of statistical methods (or at least SPC and DoE) has grown. We found that the most significant barriers to the wider use of statistical methods were insufficient resources regarding time and money, the low commitment of middle and senior managers, inadequate statistical knowledge, and lack of methods to guide the user through experimentations. We, therefore, conclude that if Swedish managers want to increase the use of SPC, CA, and DoE in their organisations, they will have to recognise that their application takes time. To increase managers' understanding of and familiarity with statistical methods and that they teach their employees in the methods are also important. Finally, we conclude that users need practical guidance through the experimentation process.