Alternatively Assessing Conceptual Learning in an Emergency Clinical Environment—A Mixed Methods Design

Modeling cognitive processes in clinical learning environments is a necessary first step towards improving learning assessment and medical practice by using an alternative assessment model. Verbal protocol and cognitive content analyses are effective methods of exploring such cognitive processes. For the purpose of simplifying the discussion, we have labeled these processes as Identification of Information, Advanced Cognition, and Medical Cognitive Action. Exploring problem solving processes with Bayesian network techniques can characterize students' dynamic learning processes quantitatively, identify differences in cognitive components at different stages of learning and better represent clinical problem solving features. We develop a hierarchical cognitive model as a cognitive assessment tool to describe the complex cognitive network relations, which can be applied to various clinical cognitive situations. The study concludes that the cognitive model was useful in identifying students' learning trajectories by representing the different cognitive features.


Introduction
An examination of medical student clinical learning through verbal protocol analysis can contribute to our understanding of the diverse aspects of problem solving (Lajoie, Greer, Munsie, Wilkie, Guerrera, & Aleong, 1995;Lu, 2007;Rose, Bearman, Naweed, & Dorrian, 2019). This study seeks to enhance our understanding of how medical students acquire competence in clinical problem solving by modeling their problem solving practices and learning trajectories. We are most interested in how medical students come to identify important information, what is an indicator of advanced cognition and what actions reflect medical cognitive actions in clinical problem solving. Three research questions are addressed: (1) How do medical students identify relevant information, evaluate evidence, and take action in the learning process in a clinical environment? (2) How can we establish a moment-by-moment model that can dynamically represent students' cognitive trajectories? (3) What are the features of student problem solving trajectories? Simulated clinical problem solving data will be used to answer these questions. Lajoie (2003) proposed that identifying expert's knowledge, skills and competence can help construct the trajectory toward competence for specific cognitive tasks. Learning trajectories have several signposts where learning transitions can take place. Implicitly, these signposts can be represented as critical cognitive components applied in describing problem solving processes (Zhang & Frederiksen, 2007;Zhang & Lu, 2014a). The learning trajectory concept reflects the dynamic nature of learning and that there are transitional processes that lead to expertise. Zhang and Frederiksen (2007) have explored the learning trajectory concept for statistics problems that require the use of ANOVA. They developed a Bayesian network model based on the cognitive model of the domain. Other similar studies can be seen in mathematical learning assessment (Zhang, 2016; and argumentative learning (Zhang & Lu, 2014b). The procedures are applied to this investigation where we analyze clinical problem solving through a verbal protocol analysis.

Cognitive Skills and Sequential Levels
For some problem solving tasks, the sequence of performing certain cognitive skills is important and consequently must be identified in order to promote learning. With respect to reasoning, Henri (1992) identified five sequential skills: elementary classification, in-depth classification, inference, judgment and application of strategies. Complex problem solving has been described in terms of the plans, goals, operators, methods and strategies needed to solve problems (Hayes, 1989;Newell & Simon, 1972). Lu (2007) has explored clinical decision-making and communication in the context of groups participating in a simulated medical emergency activity. Analyses of the decision-making activities (Werner, 1995) revealed a series of sequential components: planning, data collecting, managing, and interpreting patient data. The analysis from the communicative activities revealed a sequential pattern consisting of informative, argumentative, elicitative, responsive, and directive acts. Evidence from studies of reasoning, decision making and communication can inform future research in the area of cognition.

Cognitive Hierarchical Model
Sequential cognitive skills can be used in representing differential learning trajectories. However, as these cognitive skills cannot be directly observed, their structure can be presented in a hierarchical model (Zhang & Frederiksen, 2007;Zhang & Lu, 2014b), in which explanatory variables obtain support from evidence variables ( Figure 1). In other words, a set of explanatory variables representing a cognitive structure obtains information support from another set of explanatory or evidence variables. Horizontal nodes represent sequence and vertical nodes represent the flow of information from evidence and explanatory sources. Fig. 1 represents the relations of sequential and hierarchical features. Explanatory variable BI, B2 and B3 are sequential. The hierarchical relation has been represented, for example, from Explanatory A, Explanatory B2 to Evidence C3.

Research Designs and Paradigms
This study used mixed research design, of which there are two steps. The first step is qualitative data collection, in which cognitive task analysis method was used. The second step is to quantify the collected data in the cognitive task analysis (Annett, 2000;Chipman, Schraagen, & Shalin, 2000;Seemster, Redding & Kaempf, 2000). A Bayesian network (Conrady & Jouffe, 2015) was used to quantitatively represent a cognitive model (Almond, Mislevy, Steinberg, Yan, & Williamson, 2015), which recorded both individual student and a group of students' learning trajectories. In the mixed methods research, this design was called exploratory sequential design (Creamer, 2018;Creswell & Plano Cark, 2018;DeCuir-Gunby & Schutz, 2017), which effectively represented qualitative data collected by using cognitive task analysis or other alternative qualitative coding techniques (Miles, Huberman & Saldana, 2014).
analysis. Thus, the researcher attempted to obtain deep understanding in constructivist perspectives. When researchers moved to the quantitative phase, the post-positivism will be used in explaining the analysis results and findings. Consequently, dialectical pluralism as a paradigmatic framework is appropriate in the study to assemble both data from the cognitive task analysis and the one represented in a Bayesian network model.

Scenario and Data
The scenario was from the study of medical problem solving (Lu, 2007) and medical student learning (Zhang & Lu, 2014b), in which a clinical case was presented to medical learners. The data were a simulated data set. A 37-year-old man with a history of alcohol abuse presents to the emergency room with sudden onset diffuse abdominal pain. Eight third-year medical students participate in the study to discuss how to approach diagnosing the case. The clinical teaching session was based on a simulated scenario in the emergency room.

Cognitive Tasks and Analysis
Cognitive tasks are critical targets from which evidence variables can be extracted. The design is based on cognitive theory and research findings (Lesgold, Lajoie, Logan, & Eggan, 1990;Hollnagel, 2003;Riera & Debernard, 2003;Elm, Potter, Gualtieri, Easters, & Roth, 2003). In this study, the statement of participants problem solving strategies were qualitative data resources. Idea unit analysis (Frederiksen, 1986;Ayala, Yin, Shavelson, & Vanides, 2002) is an analytical technique for extracting variables from cognitive tasks. Idea units have been decided based on theoretical constructs which can function as both criteria for extracting evidence variable values and model components for presenting different aspects of competency (Elm, Potter, Gualtieri, Easters, & Roth, 2003;Ayala, Yi, Shavelson, & Vanides, 2002) to identify their clinical reasoning processes. Idea units connect both evidential data and theoretical constructs.

Bayesian Networks
Bayesian networks (BNs) are also known as belief networks, which are usually represented in a directed acyclic graph (DAG) to model a learning process or a problem solving process and procedure (Culbertson, 2016;Mislevy, 1994;Pearl, 1988;Tan & Tay, 2010). Koller and Friedman (2009) state that, assuming there is a class of variables which can be designated by x 1 , x 2 , … x n and C. The structure can be seen in figure 2. In such an example variable x 1 , x 2 , … x n are observed. The upper level variable C means a class. It represents a concept, which is supported by all of these observed/evidence variable x s . All of these variable x s are conditionally independent on the variable Class. Thus, the relationship of all of these variables can be described in a model which factorizes as: ( , 1 , 2 ⋯ ) = ( ) ⅈ=1 ( ⅈ | ) This model can represent the joint distribution by using a group of variables, which include a prior distribution P (C) and a set of P (X i )| C). The model indicates that a concept, which is usually a latent variable, can be supported by a group observed variable x s .

Bayesian Networks and Cognitive Features
In this study, a hierarchical Bayesian network (BN) of cognitive feature trajectories has been developed based on the analysis of clinical problem solving process in the aspect of clinical information collection and application. A BN can be used to model complicated cognitive tasks where researchers are more concerned with cognitive processes. In its graphic model, it can be described as a directed acyclic graph (DAG), and each node represents either explanatory or evidence variables and each directed edge connects two variables (Aryadoust, 2011;de la Torre & Minchen, 2014;Zhang & Frederiksen, 2007). Although explanatory variables cannot be directly observed, they can be informed by evidence variables which are obtained from the data. A BN model is instantiated by updating evidence variables.

Development of Cognitive Variables and Model
A cognitive BN is composed of a configuration of evidence and explanatory variables. Evidence variables collect evidence from cognitive tasks (Zhang, 2007;Zhang & Lu, 2014a). Explanatory variables are theoretical constructs. In this study, evidence variables represent observable judgments from simulated students' learning protocols, which are the records of students' problem solving process based the clinical case.

Definition of Evidence Variables
The cognitive model is organized hierarchically and the evidence variables at the bottom layer of the model are used to extract variable values from the data by the idea unit analysis (Zhang, 2007;Zhang & Lu, 2014b). There are ten evidence variables.
1) Identifying critical elements among these clinical Information: Students list related information which they believe are useful in the diagnosis and patient management process. The symptoms and clinical test results may provide important information about the patient. The students should carefully and systematically recognize the useful information. In addition, time is a sensitive factor in life and death situations ("I would check the vital signs". These students should be able to select and identify critical elements of the information in the time sensitive clinical environments. 2) Situation judgment: This variable describes students facing a situation where patients may have more than one problem or a set of syndromes "He recently had a sports-related shoulder injury and now has new abdominal pain". The students learn to organize critical symptoms and find relationships among these symptoms. In this variable, an inferential and logic chain may develop.
3) Medical decision-making: Students express their opinions and make a diagnosis. E.g., "I would think he would have tachycardia." 4) Evaluation: Students give a rationale for a judgment, opinion, or action. E.g., "I would ask for an abdominal X-ray to look for free air under the diaphragm, to see if he has a bowel perforation." 5) Medical metacognition: It is an advanced cognitive process which assesses and analyzes evaluations and judgments. For example: "I'm worried about the blood pressure, because I think he'd be tachycardic, and have a fever. But I do not think he'd have high blood pressure. " 6) Patient history: Students intend to collect more information about patient history. The purpose is to find evidence to support current problem solving and reasoning. E.g., "I would like to know his meds and what he took for the shoulder pain." 7) Physical examination: Students intend to collect more information through physical examination. E.g., "I would ask for blood pressure on both arms and I would do an abdominal exam looking for signs of peritonitis." 8) Doing laboratory test: Students intend to collect more information through laboratory test, such as "white blood cell count". 9) Giving first-order medication: This is a cognitive action that the medical student would take after a judgment. " … I would start him on intravenous broad-spectrum antibiotics" 10) Monitoring patient: This is a more detailed action that might include observations and control of patient situation. For example: "give oxygen..."

Development of Explanatory Variables
Explanatory variables consist of two layers in a hierarchical cognitive model (Zhang, 2007). It is more convenient to describe the explanatory variable from the very bottom level which is close to the evidence variables. As the model is completely established, the information of the explanatory variables will be transferred from the top explanatory variable to the evidence variables.

The Lower and Upper Levels of Explanatory Variables
As soon as those evidence variables are defined, the lower explanatory variables have to be developed. The lower level means that the explanatory variable is closely connected to the evidence variable; but they are not observed from the data set or they have to be updated by evidence variables. Ten evidence variables have been grouped into four categories. Recognizing

Initializing the Bayesian Network Cognitive Model
There is not much evidence of probabilities distribution in each cognitive component in the cognitive model. It is normal that any Bayesian network needs to be initialized with data which can be artificial. However, doing so does not bother the following processes. Therefore, the value of the top cognitive component is set at 0.5 as successful performance in Emergency Performance Assessment. The values of other network children (nodes), regardless of explanatory variables or evidence variables are all set at 0.7 as successful mastery of these cognitive components. The assumption of these settings is that there is no evidence to say that students express high level in Emergency Performance Assessment, so the probability of 0.5 is set in a "default" status. We have confidence to believe that students can master each sub-explanatory and evidence component at above 70% of chance after they finish a clinical cognitive process.

Simulating the Bayesian Network Cognitive Model
There are ten evidence variables which can be used to update the entire cognitive model. A random sampling method is applied to test evidence states from one true evidence to ten true evidence observations. The true status means the variable acquires a positive value based on the student's statement in the problem solving processes. In order to reduce the tedious process of the test, the random process starts from zero performance models, which means that the 10 evidence variables do not present any positive values. The student masters zero evidence variables. At the next step, the simulated data of the evidence variables demonstrates mastery of one, two and three evidence variables until complete mastery model which means the student excelled in all ten evidence variables. The mastery statuses of problem solving processes are listed in Table 1.  Table 1 shows relations of various instantiated evidence variables and explanatory variables. The more important column is Emergency Performance Assessment, which indicates the general level of the problem solving processes. When only one positive evidence is in the model, the Emergency Performance Assessment is 0.5800 and then increases as the evidence increases. When all 10 positive evidence fills in the model, the Emergency Performance Assessment value is 0.8504. The probabilistic range from 0.5800 to 0.8504 provides a scale to differentiate different cognitive trajectories.

Classification of the Students' Emergence Performance and Measures
The measures can be defined in different ways. One of the measures is based on the range of the entire probability and theory-driven categories. For example, if a researcher believes three categories are appropriate, e.g. lower, middle and high, the researcher can define the measure distances in three intervals: (a) 0 ≤ x ≤ 0.33, (b) 0.33x≤0.67, and (c) 0.67 ≤x≤ 1.00. However, the different distributions of given specific data sets are not considered in the above measures. The categories are not sensitive to the given data sets. Therefore, the categorical validity will be weak.
The second measure can be defined in terms of the practical range of each data set. In other words, we entertain the lowest and highest values of a random variable. This measure can effectively describe the classification of a set data. If there are n categories, the classification will be defined mathematically, Where X c is the interval of the classification, R p is the range of the probability values of a given data, and n is the number of classification. We assume the lower, middle, and high are appropriate classifications in our case. So here n is equal to 3.
The third measure can be defined in terms of the practical range of each data set and weighted B s . Regardless of how we consider the weights from any theory and related evidence from data-drive reports, a set of weight B s can be incorporated into the classification. Thus, the classification will be defined mathematically, X ci = B i R p /n Where i is from 1 to n.
In this study, we have chosen the second measure because this is a data-driven study; we have no theories or research evidence to refer to. Therefore, the second measure is an appropriate measure in determination of the classification.

Students' Emergency Performance Assessment Categories
There are 8 participants in the study. In Table 2, Emergency Performance Assessment presents a general cognitive feature based on these students' data. Identification of Information, Advanced Cognition, and Medical Cognitive Action represent three different aspects. The probabilities beneath the four columns of cognitive terms quantitatively describe these features. Emergency Performance Assessment can be directly used to represent general "rich" level of problem solving processes, while three sub-categories emphasize different aspects. In other words, there might be similar values of Emergency Performance Assessment for each subject, but the weights of three sub-categories are probably different. The L, M and H represent lower, middle and high, three skill levels of problem solving processes. The values are probabilities that indicate the extent to which the judgment of levels can be trusted. These levels provide suggestions to researchers and students and indicate what kinds of information and at what levels they were produced, as well as what clinical information collection and application processes are useful in a problem solving process. Based on the results in Table 3, Emergency Performance Assessment can be categorized into 3 general groups: L, M and H groups.
However, if we look at the whole patterns of the top four latent variables, students' performance scores can be classified into four different patterns, which can be seen in Table 4.

Conclusions
This study explores a cognitive process indicating how medical students recognize information, experience deep cognition, and take action in a simulated emergency medicine situation. The cognitive process describes students' clinical information collection and application process. A general explanatory variable has been defined as Emergency Performance Assessment. The Emergency Performance Assessment consists of three cognitive explanatory variables sequentially: Identification of Information, Advanced Cognition, and Medical Cognitive Action. In order to further examine the Emergency Performance Assessment, a hierarchical cognitive model has been established through the analysis of the student learning task records. The explanatory variables of the Emergency Performance Assessment cannot be observed. They are latent variables for explaining students' different features in clinical problem solving processes. Ten evidence variables are developed through the idea unit data analysis. These instantiated evidence variables are used to update the entire model.
A Bayesian network was applied to propagate the information from evidence to explanatory variables. A set of simulated network data provides a scale that allows knowing the variation of the students' levels in clinical information collection and application. Medical Cognitive Action results indicate a lower value. The other three latent variables indicate higher scores, which present four different patterns. These patterns can provide differential information to analyze different stages in a problem solving process. When facing a different task, students have to find and select related information with the clinical case. It is very challenging for students to make decisions and to take medical cognitive action. It is reasonable that students can contribute more in the emergency scenario if they have more opportunities to search for more information from available resources such as an electronic clinical library or from the clinical tutor.
In brief, the hierarchical cognitive model has effectively identified different types of students' competencies in problem solving processes. The model provides moment-by-moment differential information for both students and cognitive feature categories. The cognitive Bayesian network will be more robust in differentiating different student groups and cognitive feature categories with updating evidence by the Bayesian network learning. The cognitive construct is established by theory-driven analysis. Therefore, the construct validity will be further explored with extended clinical data.
In addition, this study takes a mixed methods design. There are two steps in this design. The first step is qualitative data collection, in which cognitive task analysis method was used. The second step is to quantify the collected data in the cognitive task analysis. Further, a Bayesian network was developed to quantitatively represent the trajectories of the student problem solving processes. This qualitative-quantitative sequential design is a practical example for medical instructors to design their students' learning and achievement assessment in medicine.

Limitations
This study had data from 8 students. The analyses and findings are not generalizable to all clinical learning environments. However, the Bayesian network model and cognitive task analysis are both good examples for medical instructors to think about how to evaluate student's learning in a complex cognitive environment.