Edinburgh Research Explorer Concerns that linger: Medical school perspectives on the UKMLA and shared assessment

,


Introduction
The General Medical Council (GMC) have progressed from indicating their "support in principle" for a United Kingdom Medical Licensing Assessment (UKMLA) to managing the development of this assessment as the gateway for graduation from UK medical schools.Their consultation phase with medical schools from 31 January to 30 April, 2017 on the principles behind the UKMLA and the student intended learning outcomes has now closed and the UKMLA is scheduled to be released in 2022 for use with UK medical students (GMC, 2015;2017a, b).The GMC's initial proposal involved the release of a three-part assessment of applied knowledge (part 1), professional and clinical skills (part 2) and fitness to maintain a licence to practise (part 3).Their intention was that UK medical students would require to succeed in parts 1 and 2 to register as a doctor and obtain a licence to practise.Registered doctors would require to succeed in part 3 as a form of re-validation.Currently, UK doctors undergo re-validation approximately every five years by means of an appraisal process.In December 2017, the GMC published their decision to postpone the delivery of part 2, with the following announcement from the Chairman of the GMC, "We are not ruling out moving to a single assessment of clinical and professional skills at some point in the future.However, we cannot begin to explore this without making sure that we first have the correct foundations in place."(GMC, 2017b) Previous plans to use this exam as an entrance exam for international medical graduates (IMGs) to practise in the UK have been put on hold in the light of uncertainy arising from Brexit about the future movement of EEA medical graduates into the UK.However, it is clear that the GMC see their existing model as "progressive" (GMC, 2017b) and that the release of a single three-part assesment for UK medical students and IMG's remains a future possibility.Such developments reside in the backdrop of long-standing debates regarding the prudence of such a decision and the closely connected idea of introducing a European licensing examination (Noble, 2008;Ricketts & Archer, 2008;van der Vleuten, 2009;Archer et al., 2016) with similar issues being raised in each case.
While a medical licensing assessment is a special case of shared assessment, in preparing UK medical students in an equitable and cost-effective way for such an assessment, the use of shared assessment more generally within undergraduate curricula is also of interest in this study.Indeed, it has been suggested that construction of a shared core curriculum across medical schools involving "consensus-built foundational learning objectives" is a viable option for preparing medical students for the US Medical Licensing Examination (USMLE) (Le Tao & Prober, 2018).This suggestion is aimed at avoiding development and maintenance of "custom but duplicate curricular content" where there are already "limited faculty and financial resources", thus "bend [ing] the cost curve for medical education in the USA".A shared core curriculum is also recognized as a possible means of addressing the "anxiety, stress, and even misguided outrage" from students on account of "perceived misalignment between what is tested on USMLE Step 1 and what is taught in the relevant courses at their medical schools" (Le Tao & Proper, 2018).The above approach to using shared assessment may be transferrable to the UK and other countries faced with similar current or future resource challenges.
Within the UK itself, considerable groundwork has already been completed in support of the development of shared assessments.During 2013-2016, the Medical Schools Council Assessment Alliance (MSCAA) common content pilot project involved the sharing of an item bank of Single Best Answer (SBA) questions for use in standard set finals examinations across participating medical schools.This successful initiative involved increasing participation on a yearly basis, up to 30 UK medical schools in 2016.Currently, all but one of the 33 UK medical schools which are members of the MSCAA both contribute to and draw from the existing MSCAA assessment bank for the delivery of summative assessments.Furthermore, new question formats other than SBA are emerging.However, it is not self-evident that use of shared assessment per se should suffice as a guarantor in uniting UK medical schools with respect to standards of minimum competency at graduation.Indeed, previous research suggests that even where medical schools share assessment questions, pass marks may differ (Boursicot et al., 2006).
With this background in view, it makes sense to step outside the constraints of formal GMC consultations to explore views expressed across UK medical schools, both on the introduction of a UK national licensing examination and on the capacity of shared assessment more generally to support uniformity of competency across UK medical schools.
Based on an online survey, I therefore provide an overview of viewpoints across UK medical schools on national assessment and shared assessment more generally.This work will serve two main interests.Firstly, it will form the basis for a follow-up study exploring to what extent medical schools see their concerns as having been addressed by the GMC further to the development and various stages of implementation of the UKMLA.Secondly, through acknowledging respondent concerns expressed through their free text comments, I will provide recommendations for good assessment practice in the implementation of national assessment and shared assessment more generally which ought to be transferrable to a wide range of academic and professional disciplines.These are important steps in promoting unity between regulatory bodies and stakeholders in the development of national qualifying examinations.Recent complaints have been made public that solicitors' views have been ignored by the Solicitors Regulation Authority (SRA) in response to their three-year corporate strategy, including in relation to the delivery of the Solicitors Qualifying Examination (Hyde, 2017).Lessons can be learnt from this case by disseminating medical educator concerns about the UKMLA and principles for addressing these concerns in advance of its delivery.
Previous work has highlighted medical educator views on the introduction of a UK national medical licensing examination.However, much of this work has comprised opinion pieces.These opinion pieces have favoured the initiative (Ricketts, 2008) or, as with a systematic review involving "highly developed countries" (Archer et al., 2016), highlighted the apparent lack of an evidence base for confirming some of the alleged benefits of the UKMLA (Noble, 2008;Allawi et al., 2015), including improved graduate competency to practise and an increase in patient safety.Other work has presented both sides of the debate based on isolated quotes from a selection of medical graduates or educational researchers in the absence of a convincing sampling frame (Rimmer, 2014).While such efforts have served to keep the debate active and highlight future research opportunities, this has been at the cost of any systematic attempt to glean views across a broad range of stakeholder universities, independently of formal GMC consultations.By contrast, the current work uses a synopsis of ideas from stakeholder medical schools to identify recommendations for good practice in the delivery of the UKMLA and shared assessment more generally.This is with the understanding that the list is not exhaustive but rather, representative of survey respondent concerns that need to be highlighted and addressed.

Conduct of Survey
Using a secure online survey system, the 27 out of a potential 34 medical schools which had previously consented to participate in a comprehensive national survey on assessment and standard setting (MacDougall, 2015) were invited, as part 2 of this survey, to respond to a matrix-style question on national and shared assessment.This question was merged with a follow-up free text question.Respondents were advised in the first instance that the principal aim of this second part of the survey was to gain an "up-to-date perspective on shared assessment and national examinations." The survey response data were collected during the period 13-30 January, 2014.The earlier part of the survey, which was carried out during the same period, involved a review of UK medical school practices in choice of question styles and corresponding standard setting methods at different stages of assessment within their undergraduate curricula.Full details of the findings of this part of the survey and the procedure for recruitment of respondents for the survey as a whole are already published elsewhere (MacDougall, 2015).Figure 1 presents the two questions (here denoted by "MQ" and "FTQ" to signify "matrix-style question" and "free-text-style question", respectively), together with the supplementary content originally provided on introducing these questions.
Figure 1.Content from survey questions on a) introduction of UK medical licensing exam and b) shared assessment While some countries have national licensing examinations (e.g.Canada and USA), there is no such arrangement in the UK.A related issue is the use of shared assessments.FTQ.Please state your views on national examinations in the UK and on shared assessment between universities.
The stems of the Likert scale questions, above were ordered as illustrated to avoid the component on shared assessment more generally serving as a leading question to the component on national examinations in particular.This was important, noting that sense of need for a common standard (or, "uniformity of competency") across medical schools may incentivize the introduction of a UK national examination (MacDougall, 2015).The 5 lines by 100 characters text box provided for the free text component (latter part of question) afforded considerable space for respondents to elaborate.

Statistical Analysis and Presentation of Data
A scatter plot of Likert scale categories Strongly disagree to Strongly agree was generated for views on the statement pertaining to introduction of a national licensing exam ("MQa") versus views on the statement pertaining to shared assessment as a source of uniformity of competency across medical schools ("MQb").
Responses on the above Likert scale were also considered separately for each of shared assessment and a national licensing examination through calculating the respective modes, frequencies, and percentages and corresponding 95% CIs.In addition, the original Likert scale response categories were merged and simplified to comprise the three categories negative, unsure and positive, thus giving rise to a derivative variable for each of MQa and MQb.Frequency and percentage data for the derivative variables were then displayed by means of a panel pie-chart and cross-tabulated by means of a frequency table for national versus shared assessment.In addition, the Rao Scott chi-square test (Rao & Scott, 1987), with a finite population correction (Cochran, 1977), was used to test for an association between views on introduction of a UKMLA and on the utility of shared assessment as presented under the statements for MQa and MQb.
All free-text response data specific to the question asked under FTQ were summarized in table form at the level of the individual respondent.This was with the understanding that the perspectives of any one medical school had the potential to influence a vast number of future stakeholders, including medical graduates and their employers and patients.As such, they were stated in the respondent's terms, inclusive of any nuances.
Table 1 displays paired frequency data for respondent views on national assessment against their views on shared assessment.These data revealed a lack of statistical evidence for an association between views on introducing a national licensing exam and on the utility of shared assessment in unifying competency across medical schools ( 2 = 0.142, p = 0.998).

Perspectives on National and Shared Assessment
For completeness, the respondent perspectives on national and shared assessment arising from the free text question FTQ, above are provided in Tables 2-4.Preference to express viewpoint as "mixed" or "neutral" rather than "unsure"; "substantial benefits": "The opportunity to identify seriously underperforming students, schools, or cohorts; "a guaranteed set of standards"; "substantial costs": "the shape of the exam would have a huge impact on medical practice, and errors in the delivery of such an exam would be catastrophic"; "units might 'teach to the test' …" 10 Unsure Cost: "superficial learning" driven by "extrinsic motivator"; benefit: "equivalence of standards" 12 Unsure Such an exam is on the one hand, defended as having the capacity to address "some aspects of public 'uncertain[t]y"' but on the other hand, criticized as having the potential to be detrimental to graduate qualities through homogenization of curricula.
11 Unsure Cost: parallel drawn between use of the National Student Survey as too much of a focus for decision making and the potential for a national exam to become "a focus of all teaching and learning".In the latter case, the school envisaged a league table as the product of a national exam which would drive teaching at the cost of "a more rounded educational approach."

Medical School Response to MQb
Perspectives on national examinations in the UK 14 Unsure School expressed full commitment to the above MSCAA project, but they were still at the monitoring stage.(MQa: Disagree) 12 Unsure "At present shared questions through MSCAA seems an appropriate next step whilst schools become confident around the standard of questions approved." 13 Unsure Cost: did not see "uniformity of graduates" as a goal to pursue but, rather, perceived the collective diversity in graduates across medical schools as an outcome to "celebrate"; benefits: favoured shared assessment for "many pragmatic reasons" Table 4. Perspectives on national examinations in the UK and on shared assessment between universities for those who responded negatively to statement "National licensing examinations should be introduced in the UK" (MQa) or "Shared assessments between universities will result in uniformity of competency between graduates of different medical schools" (MQb)

Strength of Views on the Introduction of a UKMLA and on Shared Assessment as a Source of Uniformity of Competency
Can we assume that strength of feeling regarding the capacity of shared assessment to create uniformity of competency is a good predictor of level of support for the UKMLA?The findings from data exploration (Figure 2a and Table 1) and statistical hypothesis testing suggest not.Thus, seemingly, the opportunity to achieve a common standard of minimal competency does not necessarily provide medical schools with an impetus to favour national assessment.It remains possible that those medical schools who recognized shared assessment as supportive of a common standard of minimal competency but were also unsupportive of introducing a national examination still valued the goal of achieving a common standard of minimal competency.
While the most popular choice in response to the statement "National licensing examinations should be Concern: "any resulting feedback" is likely to be used "politically" (see Table 3 for explanation); benefit: "ensuring minimal competency" 1 Strongly disagree Concerns: "Shared assessments will not produce a uniform standard set pass mark.Also, students are taking exams at different stages eg some Final MB EXams [sic] take place at the end of year 4." introduced." was "agree", notice (Figure 2a) that this choice was only marginally more popular than "strongly disagree", suggesting a strong disparity in feelings between several schools concerning the idea of a UK national examination.Indeed, Table 2 reveals that one medical school (Medical School 2) was more accepting than enthusiastic about a national licensing examination.While the above school did not deny the need for such an exam, the lingering concerns expressed by this and additional respondents which opted for responses within the range "Slightly agree" to "Strongly agree" are just as valid as those of other respondents.
Due to the comprehensive nature of the questions in part 1 of the survey (MacDougall, 2015), extending the survey questionnaire to allow an in-depth exploration of reasons for responses provided to part 2 of the survey would have been unreasonable.Instead, these responses are used in this paper as a basis for recommendations on addressing respondent concerns.

Addressing Respondent Concerns
The following recommendations emerge from further examination of free text comments on national examinations in the UK and on shared assessment between universities provided in response to the free text question labelled "FTQ", above: 1 Encourage innovation in the continuous assessment of fitness to practise, such as through a "Portfolio for Safe Practice" to enhance gatekeeping and Personal Professional Development.

2
Promote curriculum innovation to support diversity in learning styles, opportunities for student self-selection of learning and development of unique graduate selling points.This should serve as complementary to preparatory training for a UKMLA.

3
Ensure that medical school performance is evaluated by the GMC according to criteria inclusive of innovative practices in the above senses and not exclusively student performance in the UKMLA.

4
To reassure stakeholders, define and document a rigorous administrative procedure for preventing errors and maintaining security in the delivery of the UKMLA and shared assessment more generally.

5
To discourage learning and teaching "to the test", ensure that the content of the UKMLA is sufficiently flexible across separate sittings to permit intended learning outcomes rather than prior test content to drive learning and teaching.

6
Provide a prescriptive open access evaluation template, supported by statistically sound advice on interpreting reported findings, for confirming the fairness of the UKMLA and validity of UKMLA scores.

7
Ensure that the UKMLA accommodates testing and rewarding of excellence.

8
Accommodate ongoing input of UK medical educators to the final diet of summative assessment for undergraduates at their respective medical schools.9 Present a convincing strategy for addressing the needs of students who fail the UKMLA.
10 For shared assessment to be a viable option, give greater priority to blue-printing of questions.
11 To ensure equity in equipping students for the UKMLA, support sharing of assessment and standard setting in preparatory assessments.This should include striving to remove inconsistencies in timetabling of assessments across UK medical schools.
In highlighting the added value of the above recommendations, it should be noted that Ricketts (Ricketts, 2008) cites the respected Handbook of Test Development, now in its 2nd edition (Suzanne et al., 2016), as a reference point for "current standards of best practice in test procedures" in the design and delivery of "[A]ny national examination".The previous edition of this text has proved highly popular within the social sciences, particularly in relation to the development of itemized tests.The current edition seeks to complement this contribution with due recognition of technological advances and an expansion in the range of test constructs intended to meet needs within higher education and the workplace.This guide offers a wealth of information for improving test content and the interpretation of test scores.However, there is also a need to recognize and identify the unresolved tensions that exist between the GMC and stakeholders and in particular, those concerns which UK medical schools have chosen to articulate through calls for structured feedback.The utility of a portfolio of safe practice, the importance of curriculum and assessment innovation in the senses highlighted under recommendations 1 and 2, above, the call for meeting the needs of failing students (recommendation 9) and the need for resolving disparities in timetabling across medical schools to support shared assessment (recommendation 11) are examples of findings from the above list of recommendations not captured by the handbook or adequately addressed by the recent GMC report arising from the formal consultation process (GMC, 2017a).By contrast, the GMC consultation report has made a valuable contribution to raising awareness on divided opinions about the frequency of testing sites that ought to be used for Part 2 of the UKMLA, both in the case of UK students and IMGs.For example, 50% of the 32 medical schools participating in their survey favoured the choice of "a limited number of sites across the UK" while 16% and 9% favoured the remaining choices of "one UK site for all candidates" and "UK universities recognised by the GMC", respectively, and the remaining 25% provided no response.The report also highlights the challenge of extending the use of the existing MSCAA assessment bank to accommodate question styles, such as the OSCA, envisaged for part 2 of the UKMLA.
The report does not, however, provide concrete reassurance regarding attainment of recommendations 1 to 3, above.This type of reassurance would serve in addressing the concern implicit from many of the respondent comments that focus on the UKMLA could be at the cost of the life skills required to maintain patient safety and enhance graduate attributes.Equally so, it would help in addressing the concerns expressed by respondents in this study, and previously by a spokesperson for the Medical Schools Council (Rimmer, 2014), that medical schools should be permitted to maintain their sense of personal identity through fostering curriculum diversity.
Based on the response data from this study, however, the illustrations presented under recommendations 1 and 2 present opportunities for sustainability of curriculum and assessment innovation which focus on the needs of the learner rather than merely the right of each medical school to have its own identity.
If innovation in assessment is to incorporate shared assessment and maintaining flexibility in the content of the UKMLA (recommendation 5) is to extend to choice of question styles (or, types), lessons can be learnt from existing literature.Specially, a valuable literature study has already been carried out to identify new options for question styles to assess clinical reasoning, with a focus on computer-based assessment (van Bruggen, Woudenbergh, & Vos, 2012).Noting that the first component of the UKMLA, which the GMC expects to deliver in 2022, is likely to involve online assessment of the application of medical knowledge via a computer-based assessment, the above study ought to be of considerable relevance.However, as the authors note, some question styles may be better suited to assessing particular aspects of reasoning than others.Therefore, further research is needed to confirm validity of test scores, based on the new question styles, within formative assessment.Only then can decisions be made about taking forward these new questions styles to summative, including high stakes, assessment.
In response to recommendation 10, lessons can also be learnt by acknowledging the work of the Medical Assessment Alliance (MAA), which is mainly comprised of German medical schools.The MAA use a shared web-based system, the item management system (IMS) to manage "all processes of the assessment-workflow", including blue-printing (Hochlehnert et al., 2012).Blue-printing involves a cross-classification procedure whereby the competencies to be assessed in an exam are classified and mapped to specified question types.This is recognized as a source of validity through reducing construct under-representation and construct-irrelevant variance (Hamdy, 2006).The MSCAA has contributed greatly to the quality of shared assessment in ensuring extensive reviewing of each question considered for their item bank, involving all its member schools and external examiners in this process and monitoring item performance and standard setting for these questions across schools.However, they have yet to extend this rigour to incorporate shared blue-printing.Clearly, this development would enhance the quality of UK shared assessment in general.More specifically, this development would strengthen the contribution that the MSCAA is able to make to addressing recommendation 6, above in supporting the development of the UKMLA.Researchers have recommended that, "The quality of questions for applied knowledge in MCQ-based exams can be increased by using questions with unambiguous medical patterns to assess the clinical reasoning processes."and from this perspective, have monitored annual trends from 2006 to 2012 in testing of pattern recognition for clinical diagnosis in the MCQ component of the German medical licensing assessment (Freiwald et al., 2014).Where appropriate, incorporating this and other MCQ categories highlighted in the same work into the blueprinting process for the UKMLA may help in mitigating the level of inconsistency in assessment practices observed in the above study.Noting that the "[UK]MLA blueprint" is still to be agreed (GMC, 2017a), the door of opportunity is still open for learning from the MAA experience.
In highlighting these areas for future development, it is important not to lose sight of the impact of national assessment on medical students.Under this theme, an emphasis on concerns about ranking students (Noble, 2008) has possibly contributed to a failure in previous literature to highlight the value of supporting failing students.Through recommendation 9, this paper recognizes the need for greater foresight in supporting students beyond test completion and opens the way for discussion of mock assessments in preparation for the UKMLA.
These observations point to the need for the GMC to articulate their standards and recommendations for test development in a succinct manner, accessible for all stakeholders, including those unfamiliar with the nuances of the art and science of test development.For example, the intended mechanisms for maintenance of test security (recommendation 4) from the standpoint of the examinee, test developer and among others, expert reference group members who liaise with the GMC, need to be explained, documented and disseminated among stakeholders.
While strict procedures were followed (MacDougall, 2015) in the identification of appropriate representatives from each participating medical school in the survey for this study and allowance was made for these representatives to liaise with their colleagues before submission of their final responses, it must be recognized that the feedback generated is unlikely to reflect the views of all staff who deliver medical education within the respondent schools.However, the findings of this study should make a useful contribution to current understanding on medical school concerns about the UKMLA and shared assessment more generally and how these concerns might be addressed.

Conclusions
Shortly prior to the GMC's formal statement of support for introducing a national licensing examination in the UK, several UK medical schools from this study had already responded positively to the idea of such an assessment and to the claim that shared assessments between universities will result in uniformity of competency between graduates of different medical schools.Lingering concerns remained, however.In this paper, I present a call to the GMC to explicitly address these concerns and acknowledge that there is a camp of medical schools that may remain concerned about, and even strongly opposed to, the introduction of a UK national licensing exam.While the GMC have acknowledged medical school concerns in their formal consultation report, the attention to detail afforded by the free text comments from medical schools in this study (Table 2-4) ought to complement their existing feedback as they progress to "framing the next steps" (GMC, 2017a).To facilitate this process, I have made eleven main recommendations to support the successful delivery of shared assessment and in particular, the UKMLA.Based on their consultations with stakeholders (GMC, 2017a, b), the GMC point to an ongoing process for extension of UK national assessment beyond assessment of applied knowledge.In the longer term, this process is expected to have international impact through use of a UKMLA in assessment of IMGs.Therefore, I hope that publishing these recommendations will promote further discussion based on medical graduates' prior experiences of delivery of a Medical Licensing Assessment in other countries, including Canada, Germany, Poland, Switzerland and the USA, in addition to promoting more open discussion among future stakeholders.Furthermore, given concerns about the increased rate in emergence of new medical schools worldwide and the diversity in quality of medical education and student entrance requirements arising from this trend, the need for quality assurance is becoming more apparent.Thus, interest in developing national medical licensing examinations to ensure a benchmark for minimal competence at graduation is expected to grow (Swanson & Roberts, 2015).Correspondingly, the above recommendations are also likely to have a role to play in the development of non-UK medical licensing assessments.
Figure 2. UK medical school views on a national licensing examination and shared assessment compared: a) within schools and b) according to distribution of responses Description: The corresponding statements for which these views were expressed are Shared assessments between universities will result in uniformity of competency between graduates of different medical schools and National licensing examinations should be introduced in the UK.In part a) of this figure, a 45° line of perfect agreement is included as a visual aid to comparison of views for the same school across these two statements.

Table 1 .
Frequency distribution of UK medical school views on national versus shared assessment

Table 2 .
Perspectives on national examinations in the UK and on shared assessment between universities for those who responded positively to statement "National licensing examinations should be introduced in the UK" (MQa) or "Shared assessments between universities will result in uniformity of competency between graduates of different medical schools" (MQb) "it will not be able to properly pick out the students who should not progress"-those "who lack certain clinical skills that might impact on patient safety"; "[i]t is easier for an individual medical school to produce" an exam relating to "professional and safety issues" -own "Portfolio of Safe Practice" highlighted; it may be necessary for a medical school to confirm that a student is professional and safe before they take a national exam; a complementary national exam may be a viable option, the case of the "SJT" a being exemplary; would accept up to 30% of OSCE as standard (MQb: Disagree) a It is most likely that "SJT" refers to the Situational Judgement Test for UK Foundation Programme applicants.

Table 3 .
Perspectives on national examinations in the UK and on shared assessment between universities for those who responded "Unsure" to statement "National licensing examinations should be introduced in the UK" (MQa) or "Shared assessments between universities will result in uniformity of competency between graduates of different medical schools" (MQb)