Finding NILMO: Integrating Curricular Development, Technology and Educational Research

Amy J. Prunuske
Jacob P. Prunuske

The amount of information delivered in medical education curricula has expanded rapidly in recent years. New and innovative curricula must cover advances in scientific knowledge, technology, and expanded core competencies in medical education, and must do so with the same or even less curricular time than in the past. Our students must develop the knowledge, attitudes and skills to competently care for patients and populations. They must be able to address current health problems and adapt to address the new health concerns of the future. This adaptability requires our students to become adept at independent learning.

When the course director for Neurological Medicine asked us to develop a series of online modules for the course, we were intrigued by the opportunity and challenge. We believed restructuring this part of the curriculum from in-class lectures to independent learning could foster lifelong learning skills in our students. We developed seven modules that we called NILMOs (Neurology Independent Learning Modules Online).

Curricular Redesign
Our past experience with online lectures suggests that students value online curricula for its flexibility(1). Students can access online curricula at their convenience and can pause and review the information as meets their learning needs. We have training in teaching pedagogy and sought to incorporate best practices in learning theory into our curricular redesign. We developed learning objectives for each NILMO and a strategy using pre- and post-tests to measure student learning gains. Our redesigned curriculum included two learning options intended to help students achieve these objectives. The first option was for students to view an online lecture. The second option was for students to use selected online resources to answer a series of questions. Our curricular outline is shown in Figure 1.

Technology Integration
We integrated the NILMOs into Blackbag, a curricular management system developed at the University of Minnesota Medical School Duluth to organize all of the Medical School courses. Blackbag integrates course resources, assignments, objectives, and an academic calendar and is used by students on a daily basis.  Students accessed each module by clicking on the assignment in the course calendar. The students’ familiarity with Blackbag facilitated the integration of the NILMOs as a new curricular element.

We used Blackbag in an innovative way that allowed us to control student progress through the module in keeping with our curriculum and study design. In most courses, students have access to all of the resources posted for each session. In our curriculum, we wanted students to complete a pre-test before accessing resources for that module. After the pre-test, students chose to view the online lecture or complete the online assignment. Students then completed a post-test before choosing whether to complete the alternate module option. We used Blackbag to track student choices as they progressed through the modules, record performance results and collect feedback on the students’ experiences with the NILMO. The success of this project was highly dependent on our ability to work directly with the Blackbag developer to program the curricular management software to meet our educational and academic needs.

Students had one week to complete each module. Students downloaded the lecture and assignment files from Blackbag. To help students complete the assignments, we included a list of recommended resources and hyperlinks including movie clips and online simulators.  Students submitted the completed assignment to Blackbag for evaluation. Blackbag also gave us the ability to limit the amount of time the students had to complete the 5-question pre- and post-tests for each module. At the end of the week, we posted students pre- and post-test scores.  The students could use these results to help prepare for the retesting of the information on the course final examination.

We experienced a number of challenges implementing our curriculum. With the first module, students did not have enough information to make an informed decision on which module option was appropriate for them. We modified the programming in Blackbag to provide students with a brief description of each module option before they made their decision. If a student chose the assignment but then decided not to complete it, he or she could submit a blank document, take the post-test and then move on to the online lecture option. When students made this choice, we nullified the initial post-test score and counted only the final post-test score that the student completed after viewing the online lecture.

We presented online lectures in PowerPoint format with audio voice commentary. A small number of students were unable to hear the audio, despite working with the Medical Schools information technology support staff to address the issue. In an effort to manage this issue, we included a written transcript in the ‘notes’ section of each slide for students to read. We also presented one lecture as a .mp4 movie. Feedback from the students was that they preferred the PowerPoint format to the .mp4 movie because the PowerPoint format allowed greater flexibility for moving forward and backward through the material or for searching and reviewing a specific point or slide.

One limitation of using Blackbag was our inability to directly measure the amount of time students spent on the modules. In Blackbag, students downloaded the various lectures or activities and could view or access them at their convenience. We would have preferred that students view or complete the various module components within Blackbag in a way that allowed us to directly measure time on task. Students’ self-reported time spent on the NILMO on the survey at the end of the module; however, these self-reports may be subject to inaccuracies or recall biases.

We designed our pre and post-test evaluations expecting that students would use whatever resources they had available to complete the tests. Because students were completing the exam outside of the usual testing center, we assumed they would access both the module activities and other resources available to them through the internet; however, because other student exams using this format were closed resource exams, some students made the assumption that the NILMO evaluations were closed resource as well.

Looking Forward
Our students overwhelmingly chose to watch the online lectures rather than complete online course assignments. Most students cited time constraints as a reason for not completing the assignment and a few students reported feeling more comfortable with information delivered in a lecture format as reasons for this preference.  Several students commented that concepts and material covered in the NILMOs should have been presented in class, rather than the online format. These students believed that the online format should be supplementary or optional rather than used for core content. There is evidence suggesting that online lecture is equivalent to in-class lecture from a learning standpoint (2), however, our students still see the in-class content as potentially more important. We recommend this tension be explicitly addressed with students when incorporating online materials into the curriculum.

We believe that instructors must critically evaluate and assess curricular innovation. There is emerging evidence that interactive online curriculum may be superior to online lectures (3). We had hoped to compare students’ learning gains between the two NILMO learning options, however because of students’ strong preference for lectures our study lacked the power to make this comparison. For the upcoming academic year, we plan to require the students to do the assignments and test our hypothesis that completing the assignments leads to increased learning gains compared to viewing an online lecture.  We will also compare whether there is a difference in number of students completing the module, time spent on the module, and perceived value of the module.

We plan to continue exploring how instructors can use emerging technology to provide innovative active learning experiences, simulation, and audiovisual materials in a way that enhances student engagement, learning and retention of key concepts.

The University of Minnesota IRB Human Subjects Committee determined this project was exempt from review.  Study #: 1109E04785

1. Prunuske J. Live and Web-based orientations are comparable for a required rotation. Family Medicine. Mar 2010;42(3):180-184.

2. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: sound educational method or hype? A review of the evaluation literature. Academic Medicine. Oct 2002;77(10 Suppl):S86-93.

3. Wiecha JM, Chetty VK, Pollard T, Shaw PF. Web-based versus face-to-face learning of diabetes management: the results of a comparative trial of educational methods. Family Medicine. Oct 2006;38(9):647-652.


Amy Prunuske, PhD <>
Amy is an Assistant Professor in the Department of Biomedical Sciences. Her laboratory is now the classroom where she enjoys experimenting with technology and encouraging medical students to evaluate information critically.
Jacob Prunuske, MD, MSPH <>
Jacob is an Assistant Professor in the Department of Family Medicine and Community Health. He is the clinical course director for the 1st–year medical student neurological sciences course and enjoys curricular development and innovation.