Notes on the SRA report of the consultation on the Solicitors Qualifying Exam: “Comment is free, but facts are sacred”1

The introduction of change is almost universally accompanied by dissent. Psychologically we are programmed to repetition or reproduction of the familiar in preference to the novel unless (and sometimes despite) the familiar being obviously unsatisfactory. It is not surprising, therefore, that the proposals to change the method of entering the legal profession have met with opposition. This particular paper will not engage with the merits of the arguments on either side, its purpose is simply to engage with the data generated by the consultation and the reporting of that data. I have highlighted the areas in the SRA’s document that caught my attention.

Notes on the SRA report of the consultation on the Solicitors Qualifying Exam: "Comment is free, but facts are sacred" 1

Northumbria University
The introduction of change is almost universally accompanied by dissent. Psychologically we are programmed to repetition or reproduction of the familiar in preference to the novel unless (and sometimes notwithstanding that) the familiar is obviously unsatisfactory. It is not surprising, therefore, that the proposals to change the method of entering the legal profession have met with opposition. This particular paper will not engage with the merits of the arguments on either side, its purpose is simply to engage with the data generated by the consultation and the reporting of that data. I have highlighted the areas in the document produced by the Solicitors Regulation Authority (SRA) that caught my attention.
The SRA has stated the following: 2 We received 253 responses to the second consultation, plus 14 responses that did not answer specific questions and instead provided general comments. We also: • engaged with more than 6,800 people through 45 events, meetings and digital activities • had almost 4,650 visits to related website pages • had 237,000 impressions on social media.
We have heard a wide range of views on the best approach.
There has been support for an independent professional assessment in principle, including from the Law Society, Junior Lawyers Division and Legal Services Consumers Panel. There has also been general support from individual solicitors and people who are still to qualify, while research has also shown that the majority of the public would have more confidence in solicitors if they all passed the same final examination. Yet there has also been clear resistance to the SQE, most consistently from academic institutions. Many respondents agreed with some aspects of our approach, but not others. For instance, the City of London Law Society and the University of Law agreed with the idea of an independent assessment, but did not support specific details in our proposals. The most consistent theme of feedback across different groups of respondents was that more work was needed to get the detail of the assessment right.
The SRA has helpfully placed a document entitled "Consultation Responses" on the website, 3 so inquiring minds can explore the data. If you copy the index from that document and paste it into an Excel spreadsheet, the first thing you notice is that there are not 253 responses listed, as the SRA claims. The document contains only 148 responses that address the feedback questions and 13 (not 14) non-specific letters. The status of the additional 105 responses was not clear: either they had been omitted from this document or there had been a mis-count. 4 An error of this magnitude led me to wonder whether any of the other claims and data needed to be checked. So I had a quick look.
"There has been support for an independent professional assessment in principle". This round of the consultation -"A New Route to Qualification" 5asked the following questions: Questions 1-6 offered the options of a five-point Likert scale (Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree) and open text comments. Question 2a was open text and Question 7 had a yes/no option and open text comments. None of these questions ask about "in principle" support for independent assessment, they are instead concerned with the fitness for purpose of the Solicitors Qualifying Exam (SQE) and the proposed arrangements. Perhaps this issue was covered in the first round?
The first round of the consultation "Training for Tomorrow" (published in October 2016) asked the 20 questions below, of which only the first appears to offer data on "in principle" agreement. 6 By eliding the two phases of the consultation, the SRA is open to the charge of misrepresenting the data.  . . . question 1 is worded so as to be self-fulfilling. It asks whether the introduction of the SQE best meets the objectives set out in paragraph 10. However, the objectives set out in paragraph 10 already assume that the SQE will be adopted. Thus, paragraph 10 openly informs the reader that it states the SRA's "objectives for the new mechanism for assessing competence prior to qualification". The first objective also assumes that the SQE will be adopted in saying that the SRA wishes to focus its "regulatory effort more rigorously than at present on assuring consistent and comparable high quality standards at the point of admission across all pathways to qualification". Question 1 is self-fulfilling because the only way of pursuing and attaining such an objective in relation to "the new mechanism for assessing competence prior to qualification" is by adopting that new mechanism, namely the SQE. In other words, the question appears to have no purpose because it appears to indicate that the SRA has already decided that it will introduce the SQE, even though the 1. Do you agree that the introduction of the SQE, a common professional assessment for all intending solicitors, best meets the objectives set out in paragraph 10? 2. Do you agree that the proposed model assessment for the SQE described in paragraphs 38 to 45 and in Annex 5 will provide an effective test of the competences needed to be a solicitor? 3. Do you agree that all intending solicitors, including solicitor apprentices and lawyers qualified in another jurisdiction, should be required to pass the SQE to qualify and that there should be no exemptions beyond those required by EU legislation, or as part of transitional arrangements? 4. With which of the stated options do you agree and why: a) offering a choice of 5 assessment contexts in Part 2, those aligned to the reserved activities, with the addition of the law of organisations? b) offering a broader number of contexts for the Part 2 assessment for candidates to choose from? c) focusing the Part 2 assessment on the reserved activities but recognising the different legal areas in which these apply? 5. Do you agree that the standard for qualification as a solicitor, which will be assessed through the SQE, should be set at least at graduate level or equivalent? 6. Do you agree that we should continue to require some form of pre-qualification workplace experience? 7. Do you consider it necessary for the SRA to specify a minimum time period of pre-qualification workplace experience for candidates? 8. Should the SRA specify the competences to be met during pre-qualification workplace experience instead of specifying a minimum time period? 9. Do you agree that we should recognise a wider range of pre-qualification workplace experience, including experience obtained during a degree programme, or with a range of employers? 10. Do you consider that including an element of workplace assessment will enhance the quality of the qualification process and that this justifies the additional cost and regulatory burden? 11. If you are an employer, do you feel you would have the expertise to enable you to assess trainee solicitors' competences, not capable of assessment in Part 1 and Part 2, to a specified performance standard? 12. If we were to introduce workplace assessment, would a toolkit of guidance and resources be sufficient to support you to assess to the required standard? What other support might be required? 13. Do you consider that the prescription or regulation of training pathways, or the specification of entry requirements for the SQE, are needed in order to: a. support the credibility of the assessment?, b. and/ or protect consumers of legal services and students at least for a transitional period? 14. Do you agree that not all solicitors should be required to hold a degree? 15. Do you agree that we should provide candidates with information about their individual and comparative performance on the SQE? 16. What information do you think it would it be helpful for us to publish about: a. overall candidate performance on the SQE? b. training provider performance? 17. Do you foresee any additional EDI impacts, whether positive or negative, from our proposal to introduce the SQE? 18. Do you have any comments on these transitional arrangements? 19. What challenges do you foresee in having a cut-off date of 2025/26? 20. Do you consider that this development timetable is feasible? consultation paper purports to seek others' opinions on whether the SQE ought to be introducedor not. 7 Loaded or not, Question 1 asked for a yes/no response, so the data is very easy to code (+1 for yes, -1 for no, 0 for neutral or ambivalent responses, all ambiguity resolved in the SRA's favour to counter any lurking bias on my part). There were 209 responses, and as Figure 1 reveals, only 20% (n = 41) were positive.
"There has also been general support from individual solicitors and people who are still to qualify". If this statement is intended to refer to the data gathered in this consultation, the implication is that individual solicitors, trainees and students are supportive of the SQE. We might expect to see a majority of these respondents selecting Agree or Strongly Agree in response to the questions: • To what extent do you agree or disagree that the proposed SQE is a robust and effective measure of competence? • To what extent do you agree or disagree that our proposed model is a suitable test of the requirements needed to become a solicitor?
"Yet there has also been clear resistance to the SQE, most consistently from academic institutions." The implication here is that academic institutions skew the results.
Using the raw data that is available it is relatively easy to investigate this. We can identify the respondents' affiliations to see who has responded to the consultation. We can then investigate the responses to the two key questions. The majority of responses have directly referred to the five-point Likert scale (Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree) and can be coded 2, 1, 0, -1, -2 respectively. Where respondents have answered the questions without direct reference to the scale, I have coded the responses based on their tone. As an academic with a vested interest, my bias in coding has been offset by "loading" the results towards the SRA proposals: a response with a mix of positive and neutral comments has been coded "Agree"; one with a mix of neutral and negative comments as "Neutral". We can get headline results from the whole cohort of respondents and then drill down into different respondent groups to explore the SRA's claims. Figure 2 indicates that 68 individuals responded on their own behalf (anonymous respondents are treated as individuals in this coding) and there were 81 responses from groups (detailed in Table 1). Bearing in mind that there will be double and triple counting of affiliations, the groups represented up to 169,914 people. 8 It must be decided by the users of this data how weighting should be assignedall responses could be treated as equal in weight, though the 170 lawyers of Linklaters and the 70,000 members of the Junior Lawyers Division might think that unfair. The SRA is keen to imply that strong vested interestsparticularly in universitiesmight overpower the voices of smaller groups and individuals. This decision can only be made when transparent data is available.  This approximate maximum figure has been calculated using each group's own data, drawn either from their response or from their website.

The responsesheadlines
The second round of consultation has produced strongly negative responses (shown in Figures 3 and 4) to the SQE both as a "robust and effective measure" and as a "suitable test", with two-thirds of respondents disagreeing or strongly disagreeing. This is not simply in terms of greater numbers of overall negative responses, but also in terms of the strength of responsenegative respondents were twice as likely to strongly disagree rather than to disagree, whilst positive respondents were more likely to agree rather than to strongly agree. The balance of feeling is weighted very much against the SQE from this data. The SRA conflates "support for an independent professional assessment in principle" with "support for the SQE" which is a stretch, since specific questions about the SQE were the subject of this consultation, not the assessment principle. There also remains an important question about the meaning of the neutral responses. I have not done a full qualitative analysis but my impression from the coding process is that the 6% 16% 10% 22% 46% strongly pro (n=9) pro (n=23) neutral (n=15) anti (n=33) strongly anti (n=68) number of responses where the message was "we do not oppose an independent assessment in principle but the SQE is not it" occurred significantly more frequently than the two examples cited above by the SRA. There are supporters (22% and 24% respectively) and the impression from the SRA document is that they are practitioners, who will be employing solicitors using this route and aspiring solicitors themselves. Figures 5 and 6 show the splits by respondent type in answer to the question about the SQE's suitability as a test of requirements. Figure 5 demonstrates that, while academics are mostly opposed, most other groups are more evenly split. Figure 6 removes the pesky academics, institutions and academic associations.
Students and traineesthe relatively small group of aspiring solicitors in this consultationare evenly split pro and anti the SQE. In contrast the employers are more anti: scepticism about the SQE is indicated by the majority of solicitors' firms who responded and more than half of individual legal professionals. Law firms, as significant arbiters of the SQE as currency in the jobs market, are distinctly underwhelmedonly just over a quarter (n = 6, 27%) of firms are positive about the robustness of the SQE and more than two-thirds (n = 15, 68%) are not. A summary of the claims, fact checking and indicative findings can be found in Table 2.
Addendum: what if none of the 105 missing responses oppose the SQE?
It's likely from a mathematical standpoint that respondents missing from consultation data would be reasonably evenly distributed across groups and points of viewin that case the shape of the pie charts and clustered bars wouldn't change that much. However, perhaps the respondents who did not wish to share their data were all much more positive about the SQEwhat might that mean? Figure 7 shows that if all the missing 105 were either neutral, pro or strongly pro, the SRA would still only be able

Status of non-included responses challenged
Support for an independent professional assessment in principle In principle question not asked in second consultation.
There is no evidence from the consultation for the SRA's claim that there is "in principle" support from more than a small section of respondents.
First consultation question 1 is problematic. Even so, responses indicate that only 20% of respondents support the SQE. General support from individual solicitors and those about to qualify Around half of "aspiring solicitors" and the majority of individual practitioners oppose the SQE.
When this is added to the majority of law firms rejecting the SQE, support from those in and about to join the profession is not evidenced. Clear resistance to the SQE, most consistently from academic institutions Clear resistance to the SQE is demonstrated by members of all groups of respondents apart from the three non-university training providers.
Resistance may be strong amongst universities but as other groups are also resistant, it would be difficult to attribute resistance just to protectionism. to claim a 46/32% majority. Of course, if it releases the numerical data, then we will be able to work it out for ourselves.

Disclosure statement
No potential conflict of interest was reported by the author.