Live-Problem Project v. Client-Based Project: Which Is Most Effective for Perceived Learning of MBA-Level Marketing Concepts?

We utilize a research design of the same professor teaching consistent course material across a core MBA marketing course to test whether a live-problem project (LPP) or a client-consulting project (CCP) is perceived by students as more effective for learning fundamental marketing concepts. Using teaching evaluation data, MBA students reported higher perceived learning from classes featuring a LPP versus those featuring a CCP. We speculate that LPPs may be more appropriate for teaching core marketing classes and CCPs for more advanced, integrative classes, such as the MBA capstone. Study limitations and future research issues are discussed.


Introduction to the Problem
Experiential education has received increasing attention as the pedagogy of choice for MBA programs eager to engage with the messy reality of management (Datar, Garvin & Cullen, 2010). Experiential education is touted as one antidote to lack of integration between knowledge, know-how and self-awareness that plagues MBA programs and limits the effectiveness of MBA graduates (Datar et al., 2010), especially, perhaps, in the more applied and judgment-oriented disciplines such as marketing (Stern & Tseng, 2002). The rationale is that by working on real-world problems in real time, students will be more motivated, will have the opportunity to integrate theory, practice and self-knowledge, and will gain an appreciation for decisive influence of context (Datar et al., 2010;Moldoveaunu & Martin, 2008).
In practice, MBA marketing professors deploy a variety of problems to stimulate learning. These include historical case studies, simulations, company challenges (such as live case studies), live problems (e.g., live data but no client), client-consulting projects, and internships (Karns, 2005). There are a number of articles that argue that students learn more analyzing historical case studies than they do through lectures (Barnes, Christensen & Hansen, 1994), and several more that suggest that various kinds of live, problem-based learning opportunities (live-problem projects, client-consulting projects, class-based businesses, internships) provide even richer learning opportunities than do historical case studies (Karns, 2005;Fletcher-Brown, Knibbs & Middleton, 2015). There are, however, few studies that have attempted to determine the conditions under which various types of live, problem-based learning options might be most effective.

Relevant Scholarship
There are many ways to teach MBA marketing classes, but as programs have shifted to student-centered learning approaches, there has been a corresponding shift to more active, interactive and experiential methods (Smith & Van Doren, 2004), a shift that has been welcomed by MBA students (Karns, 2005;Li, Greenberg & Nicholls, 2007). To many faculty and students, active, interactive and even experiential are synonymous with the case study method, and indeed this is one method for increasing classroom engagement and bringing problem-solving to the center of the learning process (Barnes et al., 1994;Webb, Gill & Poe, 2005). Although well established as an active learning method, the case study approach has been criticized for assuming that students can generalize from case to case without an underlying theoretical perspective (Grey, 2004); and for teaching managerial problem-solving without reference to the larger context (Bridgman, Cummings & McLaughlin, 2016). Hmelo-Silver (2004) describes problem-based learning as a collaborative process in which groups of students strive to solve a complex, unstructured problem that has no one correct answer -and in the process learn about framing problems, identifying and learning skills and concepts required to solve the problem, and the value of reflecting on the learning process. Savery (2006) reviews the literature to provide a useful summary of problem-based learning characteristics: Problem-based learning involves more or less unstructured problems with meaningful implications in the real world, collaboration, integration of concepts and tools from across disciplines and student leadership in the learning process. Common examples of problem-based learning include what we call live-problem projects, such as analyzing a current problem using public data, engaging in a simulation using live data (Jennings, 2002), or operating a small business (Daly, 2001).
Problem-based learning also includes projects that involve client firms -for example, building marketing plans for start-up firms (Kennedy, Lawton & Walker, 2001) or solving problems for client firms (Bove & Davies, 2009). Proponents of client-based, problem-based learning argue that such projects align with students' desire for challenging, enjoyable and real-world learning opportunities (Karns, 2005); provide the practice and real-world feedback that enhance skill development and retention (Culpin & Scott, 2012); enhance the soft skills of communication and client engagement (Bove & Davies, 2009); improve work readiness (Fletcher-Brown, et. al., 2015); and bolster professional identity (Ewing & Ewing, 2017). In short, live, problem-based learning opportunities offer a rich learning and skill-development opportunities for MBA students -perhaps especially when such problem-based projects involve direct interaction with clients. Most relevant for our study, we describe and distinguish live-problem projects (LPPs) versus client-consulting projects below (CCPs): LPPs, such as simulations based on real-time data, writing marketing plans for existing public companies, analyzing firm performance using public data and conducting marketing interviews and surveys concerning a current market trends -unfold with little or no client interaction (Jennings, 2002). LPPs start as unstructured questions about an unknowable future and must be wrestled into a tractable project by students, advised by faculty. Although work on real problem and a real company can be quite interesting, students are under no illusion that their research will be used or recommendations implemented by the focal firm.
CCPs involve firms in defining current or potential business problems for students to work on and in providing feedback to students as they work (Fletcher-Brown, Knibbs & Middleton, 2015). That is, the clients define problems, provide resources (data, introductions, fees), remain involved throughout the process, and expect an evidence-based solution to emerge as the students work through the problem in real time (Bove & Davies, 2009). The personal connection with clients, immediacy and high stakes combine to make CCPs quite engaging; there is every chance that research findings will be used and recommendations implemented. At the same time, the projects tend to be unstructured and ambiguous because, no matter how sophisticated the effort to structure the client's concerns, clients will change their perspectives and priorities based on often hidden firm dynamics (Kennedy, Leighton & Walker, 2001).

Research Question
In this study we compare the influence of two high-cognitive demand learning opportunities -LPPs and CCPson students' perceived learning. Because literature suggests that student motivation should be higher with client-based projects (Ewing & Ewing, 2017), we would expect students to report a more positive perception of a learning experience featuring a CCP. At the same time, extreme ambiguity associated with CCPs (Zsidisn, Hartley & Collins, 2013) may well overwhelm the professor's ability to provide adequate theoretical rationale, and so undercut the expected motivational advantage of CCPs -making LPPs the more effective choice. Given the lack of prior empirical research, we asked the following research question: Is there a significant difference in perceived learning for Core Marketing MBA classes organized around a live-problem project (LPP) versus those organized around a client-consulting project (CCP)?

Method
To compare the perceived effect on learning of LPPs versus CCPs, we compared the experience of students who took the required introductory marketing strategy course from the same professor midway through the curriculum of their MBA program at a common university.

Setting: MBA Core Marketing Course -Distinguishing LPPs Versus CCPs
The setting was an MBA program in a public university in the Mid-Atlantic United States. Although the study spans three years, the MBA program's selection criteria did not change during the period of the study, resulting in a relatively consistent student profile. The students studied were drawn from two types of the MBA: the full-time or Global MBAs (GMBAs) and part-time or Professional MBAs (PMBAs). The PMBA students took one or two classes/semester in the evening and the GMBA students took at least some classes during the day, but the competencies guiding the curricula were identical and the courses and professors nearly so. The main differences between the programs were that the full-time students (GMBAs) took their core classes as a cohort, received some additional professional development, and were required to attend two overseas immersions. GMBA and PMBA took the same electives in mixed classes.
The focal course was the core marketing course, designed to teach all MBA students the fundamentals of marketing. The course was organized around the writing of a strategic and tactical marketing plan for a firm facing a real-time marketing challenge. In all classes, the marketing plans were written by teams organized on the first day of class, and the weight assigned to the group project was the same for both LPP and CCP: 50%. The plan-writing process was carefully spelled out and used not only to guide the project but to apply and reinforce key concepts. The instructor's syllabus remained largely unchanged during the period of this study; the only significant difference between the students' experience was whether the focal project was for a client or not -whether the focal project was a live problem project or a client-based consulting project. A few of the PMBA marketing classes studied were delivered in a hybrid format in which in-person and WebEx class sessions were taught in alternate weeks, but this did not change the syllabus or project requirements.
The LPP and CCP experience differed in four important ways. First, for LPPs, students identified a marketing challenge faced by an existing public firm (or nonprofit with sufficient publicly available data) and built the research-based marketing plan without direct contact with the focal firm. For CCPs, students conducted the market research and built the marketing strategy for a highly engaged client firm. Thus, CCPs were defined in negotiation with clients, and client requests had a large influence on the relative proportion of attention applied to various parts of the marketing planning process. As a result, in the CCP classes, different teams spent more or less time on different parts of the plan, whereas in LPP classes, teams moved in lockstep through the planning process. Second, although teams working on LPPs were permitted to contact the company for inside information not available through public sources, there was no requirement to do so and many chose not to reach out. In contrast, the CCP teams met with their clients regularly throughout the semester -and received guidance and distraction in equal measure. Indeed, although every LPP and CCP project varied in terms of marketing challenge addressed and research conducted, CCPs also varied due to differences in client helpfulness in providing data, acceptance of class constraints, etc. Third, CCP project teams were guided by seasoned executives hired as adjunct professors (called project executives or PEs) who also provided guidance and distraction in equal measure. Fourth, clients paid fees to support PEs and, in part because of the fees paid, the clients expected to receive a commercial-grade deliverable product by the end of the class. Although encouraged to meet a commercial standard, there was no pressure for students engaged in LPPs to achieve any particular standard (beyond what was necessary to pass the course).

Sample
The samples analyzed were drawn from two closely related MBA programs, the Professional MBA (PMBA) program, n = 258 students, and the Global MBA (GMBA), n = 58 students. The data were drawn from end-of-course teaching evaluations submitted electronically and anonymously to the University's Office of Institutional Research and Assessment. We examined all evaluation data collected for the marketing course during a three-year period, Spring semester 2015 through Spring semester 2018, including fall, spring and summer semesters. The same instructor taught all sections sampled.

Measures
The analysis relies on five measures -a two-item perceived learning scale and dummy variables noting whether a student's response was associated with a LPP or a CCP; the PMBA or GMBA; a particular semester; or an in-person or hybrid delivery method. To add further richness to the quantitative measures, we also downloaded and analyzed the open-ended comments submitted as part of the student evaluations.
Dependent Variable -Perceived Learning. Two teaching evaluation items from the end-of-course evaluation were used. The items were: "the course increased my ability to analyze and critically evaluate ideas, arguments, and points of view," and "I learned a great deal in this course," both answered using a five-point response scale where 1= strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree. By using these particular items, we sought to focus on students' perception of learning not their thoughts about the professor's performance. In so doing, we attempted to sidestep some of the critiques of the validity of student evaluationssuch as concerns that workload, perceived difficulty or grading standards might affect the evaluation of teaching effectiveness (Clayson, 2009). There was a strong and statistically significant correlation r(347) = 0.84, p < .01 between the two items. Combining the items into a single, two-item, perceived learning scale was supported by a strong reliability estimate (coefficient alpha) of 0.91 (Nunnally, 1978). The scale mean and standard deviation were, M = 4.08, SD = .99. The relatively larger standard deviation across individuals suggests variance not only between but within teams. Independent Variables -LPP or CCP. CCP was set at 1, and the n was 191. LPP was set at 2, and the n was 123.
Control Variables. To control for cohort effects (either due to program or year/semester) and for teaching modality effects (hybrid v. in-person), we noted the MBA program (PMBA or GMBA), semester taken (fall, spring, summer), whether or not a particular section was taught in an in-person or hybrid format, and class size. MBA Program, PMBA = 1 (n = 258), GMBA = 2 (n = 58). The professor and course taught were common across both PMBA and GMBA students, but the GMBA learned as part of a cohort and tended to have less work experience than the PMBAs. Semester course taken, 1 = Spring (n = 192), 2 = Summer (n = 45), 3 = Fall (n = 77). Holidays and graduation schedules resulted in small differences in total class time in spring, summer and fall semesters, but the course covered the equivalent material in the same order and essentially at an equal pace in all variations. (Summer courses were approximately the same length as fall and spring semester classes, and not compressed as is often the case.) The mean class size for each section was 34 students, with a range from 10 to 49. Delivery Model, 1 = hybrid (n = 105), 2 = face-to-face (n = 246). All course sections met once per week for two and one-half hours, but in hybrid format courses, the location of delivery alternated between a physical and a virtual classroom (half of the classes in each setting). The professor made every attempt, in syllabus, materials covered, even exercises, to make the hybrid class identical to the in-person version.

Data Analysis
Using listwise deletion of data, the data set used for analysis was n = 314 across all variables. An independent samples t-test was used to test the research question. After testing the research question, additional analyses, using independent samples t-tests, correlation, and analysis of variance (ANOVA) were used to test the impact of the background variables on learning outcomes. Finally, to facilitate comparison across variables, we also used hierarchical regression analyses to combine all variables into a single model. For this analysis we combined the fall and summer classes into one category to create a more balanced distribution of spring (n = 192) versus fall and summer (n = 122).

Results
The research question asked was: Is there a significant difference in perceived learning for Core Marketing MBA classes organized around a live-problem project (LPP) versus those organized around a client-consulting project (CCP)? We addressed this question using both t-tests and hierarchical regression. Using an independent samples t-test, the perceived learning mean for LPP classes was 4.22 (n=191) versus the perceived learning mean of 3.80 for CCP classes (n = 123). The independent sample t-test result was t(214.53) = 3.54, p < .01. Thus, students taking LPP classes reported higher perceived learning than those taking CCP classes. Cohen's d = .42, which is a medium effect size for this significant difference (Cohen, 1988).
Looking at the perceived learning means for other group comparisons, there was no difference on perceived learning for hybrid (n =105, M = 4.04) versus face-to-face (n = 209, M = 4.06) classes, t(312) = -.14, p > .05. Similarly, there was no difference on perceived learning for related to the semester in which the course was taken, i.e., There was a significant negative correlation between section class size and perceived learning, r(312) = -.23, p < .01, suggesting that students perceived learning more in smaller classes. Using an independent samples t-test, PMBAs' perceived learning was significantly higher than GMBAs' perceived learning: The perceived learning mean for PMBA students was 4.13 (n = 256) versus the perceived learning mean of 3.73 for GMBA students (n = 58), and the difference was significant t(75.03) = 2.41, p < .05. Cohen's d = .28, which is a small effect size for this difference. This suggests some difference between the GMBA and PMBA experience -a difference worth exploring in future work.
The hierarchical regression analyses presented in Table 1 provides further insight into the findings.
Step 1 modeled the three control variables and showed that these three control variables explained 3% of the variance in perceived learning. Of the control variables, only the type of MBA program had a significant impact on perceived learning, with PMBA being significant, b = -.45, t(310) = -2.77, p < .01. (Because the PMBA was coded as 1 and the GMBA as 2, the negative coefficient indicates that PMBAs rated the perceived learning higher than did the GMBAs.) In Step 2, we added the CCP/LPP variable, which improved the variance in perceived learning explained by an additional 4%, for a total of 7% explained (an improvement that is significant at the .001 level). The type of MBA remained negative and significant (as above), and the LPP variable was significantly and positively associated with perceived learning, b = .40, t(309) = 3.50, p < .01. Taken together, these small (R squared of 0.07) but significant findings reinforce the insight that students in the core MBA marketing course perceive more learning from an LPP-anchored course than they do from a CCP-anchored course. The open-ended comments from students shed some light on why there was more perceived learning reported for LPP classes than for CCP classes. Typical LPP comments emphasized the close connection between the real-world problems and the concepts covered in the course: "I liked the practicum format because it was helpful in applying learning from the readings to a real-life situation." "The real-world experience and analysis of a real company were really helpful for how to engage in market research tactics, analyzing financials and other components that go into marketing." Finally, "The beginning-to-end project was extremely beneficial and allowed me to apply learning directly through a logical progression." In contrast, comments from CCP classes complained of friction between the problem and the intended learning. "This should not involve a live client problem. It is too important of a class to have core marketing concepts take a back seat to the demands of a client." But also, "I think that you are too focused on the class itself. We were assigned these projects with real clients, and I believe that they should be put first instead of classroom objectives." In short, while some preferred to focus on the client and some on the course concepts, the underlying sentiment was that, "There seems to be some disconnect between what's involved in our marketing project and what we are expected to learn through this class." Finally, "Many times, the class concepts did not align with where we were in our client projects." Collectively, these comments suggest that students perceived the CCP as competing with the class prioritiesperhaps in part because the client's interaction and demands competed with the faculty member's class structure, thus undermining the theoretical concepts put in place.

Discussion
In this paper, we explored the question of whether there is a significant difference in perceived learning reported by students in a core MBA marketing course organized around a live-problem project (LPP) versus one organized around a client-based problem (CCP). To our knowledge, this is the first study to compare empirically the effect on perceived learning of a live-problem project versus a client-based project. As a first study, we recognize the exploratory nature of our findings and the need for future research to provide further testing. One important contribution made by this paper is the empirical finding that students taking the MBA fundamentals of marketing class perceive that the live-problem project approach helps them learn more marketing concepts and skills than does the client-based project approach. A medium effect size was found for this significant difference (Cohen, 1988). The ability to measure perceived learning across the same class, taught by the same teacher, in the same MBA programs, at the same university helps to control for various alternative explanations, such as differences in professor, course design, program quality or student quality. Although the findings of this exploratory analysis are modest, they suggest that the problem-based experiential learning modality chosen for a class truly does make a difference in perceived learning. While the empirical inquiry is focused on only two experiential learning modalities, our hope is that these findings contribute to an expanded discussion concerning the most effective deployment of experiential learning in MBA programs in general and for the teaching of marketing in particular.
The comments made by students about their learning suggest at least one reason why the choice of experiential learning modality might make a difference in perceived learning. The students comment approvingly on the close fit between the live problem projects and the course concepts covered -and note with disapproval the poor fit, even conflict, between the demands made by clients and the need to cover certain course concepts. The desire for a close fit between the concepts and the project is consistent with adult learning research that shows that students learn most effectively when course concepts are carefully arranged to deliver the right level of cognitive demand at the time when the students are ready to absorb and profit from the challenge (Krathwohl, 2002).
Further complicating the CCP experience is the use of project executives (PEs) to help manage the projects, to ensure that clients receive commercial-grade deliverables, and to help students translate theory into practice. As much as they help with learning and project quality, the PEs become one more voice in the learning process and often a filter concerning which concepts are or are not reinforced. Their insights do not always align with those of the professor, and when PE's lack of familiarity or comfort with concepts, they may not reinforce core concepts with students. The result can be further deviation from the intended arc of learning, and especially a lack of clarity, precision and consistency that have been shown to make feedback effective (Datar et al., 2010). Looking ahead, it would be helpful to explore in more depth the benefits and costs of sharing course management with experienced project executives.
One possibility worth exploring in future research is whether the relative weight of motivation and engagement versus structure and safety does and should shift over students' MBA careers. Could it be that more structure and safety are appropriate earlier in the program and less structure and safety later? Said another way: Could it be that the power of motivation and engagement are realized only when students have gained enough mastery, through structured learning and clear feedback in a safe context, to accept, even embrace, the cognitive demand of evaluation and creativity when challenged by the prospect of meaningful work for a live client? Such a finding would be consistent with the logic of Bloom's taxonomy (Krathwohl, 2002) and with the research that emphasizes the developmental nature of learning. Practically, we wonder -and plan to explore in future workwhether, as students' progress through their studies, they become more open to engagement, ambiguity and challenge, and less dependent on structure and support. Our speculation, to be explored, is that case studies might be most effective early in the MBA program; live-problem projects most effective in the middle of the MBA program (where the marketing class studied is situated); and client-based projects might be most effective at the end of the program, i.e., capstone level.

Limitations
As all studies do, this research has several limitations. One is the threat to internal validity, i.e., other explanations for the results found (Stevens, 1996). Several prominent factors which jeopardize internal validity include: history; maturation; subject selection bias; social interaction; and instrumentation (Stevens, 1996). As an example of history, the professor's teaching could have changed over time (e.g., becoming more comfortable with course material, differential emphases on concepts), despite the common course syllabus. In terms of maturation, we don't know if the professor unintentionally communicated greater enthusiasm for one type of problem-based learning approach over the three-year course of this study. Looking at selection bias, the MBA students had to take this required marketing course, regardless of their interest in marketing as a major. In terms of social interaction, we don't know how the team structure could have affected the study results, for example perhaps students in the LPPs got along better versus those in the CCPs which then affected their perceived learning. The negative correlation between class size and perceived learning, and the differences in perceived learning across the PMBA and GMBA classes (in the t-test and regression analysis) further indicate that group dynamics can affect learning outcomes. Finally, instrumentation is an issue because only a two-item teaching evaluation scale was used to measure perceptions of learning, i.e., "the course increased my ability to analyze and critically evaluate ideas, arguments, and points of view," and "I learned a great deal in this course," Both of these items are very general and do not specifically compare the learning achieved using an LPP versus a CCP.
In addition, this teaching evaluation data was only measured once and at the end of the course. Clearly more specific measures comparing the learning achieved using these two different experiential-based approaches is needed, as well as measuring a professor's teaching style, and group cohesion, in future research to help control for internal validity threats.
A second limitation is the general research design, i.e., a one-course, one-faculty-member, one-university design, which limits the generalizability or external validity of results. This limitation could be addressed with additional research that compared problem-based learning approach results across several universities, programs, subjects, professors and time periods. Ideally, external funding, e.g., a grant, could greatly facilitate this effort.
Third, the exploratory quantitative analysis was limited by our reliance on teaching evaluation data. For example, there are no allowances made for students' preferred learning styles (Ackerman & Hsu, 2011). Further, the student feedback form process neglects several useful pieces of data, such as response rates and student demographics. Such demographics, e.g., gender, age, work experience, should be controlled for in future studies. Finally, the four assumptions made for differentiating the LPP versus CCP as noted in the Methods were not are verified by measuring student perceptions, since these were not collected as part of the teaching evaluation.
Fourth, there remain questions about the strength of the correlation between student evaluation scores and learning, especially in business courses (Clayson, 2009). Collectively, these limitations affected the amount of perceived learning explained. Measuring learning not only at an individual level, as done here, but using other measures, e.g., peer or team-based learning; could lead to a more robust set of findings. Future research should ask students about their perceptions of the cognitive demands placed on them, including engagement, challenge and ambiguity, as well as perceived company involvement, problem ambiguity, and motivation, and make comparisons across different experiential modalities. Measuring type of organization, type of client problem, amount of client helpfulness in providing data, and the level of client flexibility in understanding student constraints, would additional richness to the research.

Conclusion
Identifying problems in the market in addition to research, analysis, strategy and tactics planning represent important concepts taught in a core MBA marketing class. Regardless of technological changes, e.g., using Zoom versus WebEx for online learning, experiential team-based learning is often a critical component of such core MBA marketing classes. We found initial support that a live-problem project (LPP) approach helped MBA students learn more marketing concepts and skills than did a client-based project (CPP) approach. However, this exploratory, empirical comparison of perceived learning associated with LPPs versus CPPs also leads to many unanswered questions, as noted, to explore in future research. We hope this study begins to deepen our collective insight into, and will stimulate further research about, how best to deploy problem-based experiential modalities to help adult professionals learn marketing, and more generally, effective business management.