The Covid-19 pandemic has left an indelible mark on the education system, causing a fundamental change in the way we teach, learn and assess. This unprecedented situation accelerated the adoption of education technology and technology-enhanced assessment.
Institutions had to move abruptly from in-person to online classes and to rethink their assessment practices, embracing technology to try new things, creating new practices or emphasizing existing practices.
There is a lot to learn from that growth in assessment innovation.
Informed by existing literature, this small-scale study seeks to explore the opportunities (with examples mainly from Health Education Professions) and challenges of technology-based assessment in the current educational landscape.
Opportunities
Programmatic assessment
COVID-19 compelled institutions to depart from the conventional high‐stakes, summative examination model and explore alternative means of demonstrating their students’ progress. Many schools moved toward a more programmatic approach to assessment with technology making it easier for schools to carry out remotely low-stakes assessments, optimizing feedback delivery for both learners and educators.
Programmatic assessment is not a new practice that emerged out of the Covid-19 crisis but its implementation had faced many obstacles in the past due to accreditation practices and an emphasis on validity and reliability issues (van der Vleuten 2016).
Programmatic assessment that makes use of multiple low-stakes assessments across a course provided an effective alternative to the conventional end-of-course, end-of-year high-stakes assessments in a time of crisis.
Any single assessment has such severe limitations that any single is really no measurement” (van der Vleuten 2016)
Programmatic assessment enables schools to move away from end-of-course individual measures and drives learning in a meaningful way. It combines both assessment for learning and assessment for decision making and the trend towards programmatic assessment as a result of the Covid-19 pandemic presents tremendous educational opportunities.
In programmatic assessment, any assessment is seen as only one data point, “individual data points are maximized for learning and feedback value”, and high-stakes decisions are taken on the basis of many data points (Vleuten, 2012), combining assessment information across methods.
“Any single assessment has such severe limitations that any single is really no measurement”, this leads to the question: if we can’t have it in a single measure, can we then have it across measures” (van der Vleuten 2016).
Low stakes assessments have the advantage of providing actionable feedback to learners and data to inform faculty and curriculum directors where students are struggling with particular concepts (Sambell et al. 2012). Programmatic assessment overcomes the negative effects of end of year high-stakes assessments or barriers: strategic gaming, unhealthy competition and a predominance of book study, over immersion in clinical environments (Ryan, 2023), while educators get very little information about the learners.
As education moved online, technology provided a great support for the delivery of assessment and the provision of feedback for learning. Using technology, if fit for purpose, can enhance learning in assessment and feedback processes (Deeley, 2017), freeing time up for faculty to focus their time on learner support. TEA can provide significant benefits in collecting data and tracking learner progress —supporting longitudinal learning and continuous assessment and enabling “frequent look and rapid remediation” (Ricketts and Bligh 2011).
Institutions have been using a wide range of technologies, from online quizzes to more engaging activities such as student-generated YouTube clips to demonstrate an understanding of clinical and scientific concepts.
Technology has also enabled the delivery and automated marketing of alternative knowledge testing formats including very short answer questions, allowing candidates to answer using free text and the use of machine learning algorithms to understand most of the market (Sam et al. 2018).
Online OSCEs
Objective Structured Clinical Examinations (OSCEs) are in-person examinations that measure clinical competencies by rotating students through multiple stations, each of which usually lasts between 6 and 15 minutes, each station assessing a core skill or combinations of skills.
With social distancing measures during the Covid-19 pandemic, technology has enabled institutions to conduct OSCEs entirely online, using flexible and innovative solutions to reduce the disruption in examination of clinical competencies.
In Finland, a modified online OSCE was organized simultaneously in all dental institutes, in April 2020, as a replacement of the live national OSCE, using the Moodle virtual learning environment. The traditional OSCE format had to be restructured. Students could not be assessed on the “show how” but only on the “know how” level of Miller’s pyramid. The clinical tasks were modified to be problem-solving questions related to the clinical work to provide clinical application scenarios which are of relevance in clinical practice. The positive feedback towards the online examination encourages including an online examination to complement the traditional OSCE in the future (Hytonen et al. 2021).
Same success was observed in the United States at Harvard School of Dental Medicine (HSDM) where an online OSCE was administered using the Zoom platform and its breakout rooms feature which allows private minisessions (one‐on‐one) with one student and one examiner. Six minutes were allotted per station for oral examinations based on the clinical case that students were given before the session. When the allotted time had passed, students were moved to the next room. Examiners graded students on Qualtrics (Qualtrics, Provo, UT) or on hard copy.
Most students thought the online OSCE was similarly successful to traditional OSCEs, half of student respondents thought that even future OSCEs should be performed online. Examiners also felt the OSCEs ran well overall. Considering the overall success of the online OSCE and positive feedback from students and examiners/staff, there may be value in moving traditionally in‐person assessments online, even post‐pandemic. (Kakadia et al. 2021)
At the University of Almeria with nursing students enrolled in a clinical placement module, a virtual OSCE was designed, using the Genially platform, an online platform for creating interactive content, to set up an assessment as a story-telling game-like model (Roman et al. 2022). Serious game-like models in online OSCE “appear to be an appropriate alternative assessment method for face-to face approaches, helping students to remove perceived barriers of face-to-face” (Roman et al. 2022). Online OSCEs employing game-based models create a realistic assessment environment where students can manage patient details at their own pace, mitigating the nervousness associated with the scrutiny experienced in face-to-face assessments. Utilizing storytelling game-like models in online OSCEs appears to enhance learning and reflection, aspects that participants may not perceive as prominently in face-to-face settings (Roman et al. 2022).
A UK pilot study, carried out in 2022 to determine the feasibility of virtual OSCE teaching when preparing for summative OSCEs, reported that most participants in the virtual OSCE sessions reported an improvement in their confidence in history-taking, communication and data interpretation skills. The majority of students and examiners also reported that they found virtual OSCE sessions to be as engaging and as interactive as in-person teaching (Grover, 2022).
The success of such initiatives encourages the adoption of a hybrid online-onsite model as an alternative to in-person OSCEs (Hara, S., et al. 2023), which could help prepare for future outbreaks of infectious diseases and address the challenges associated with traditional, in-person OSCEs that require significant human resources.
While the implementation of Hybrid OSCEs comes with its own set of challenges, such as ensuring the reliability of virtual assessments and maintaining security and integrity, the benefits in terms of flexibility, scalability, and adaptability make them a promising option for the future.
Open-book examinations
The coronavirus pandemic gave educators the opportunity to explore the use of online OBEs in medical education as the feasibility of holding secure closed-book examinations was compromised (Zagury-Orly et al. 2021).
Advocates for CBE generally contend that CBE requires more effortful retrieval from memory, which benefits learners’ long-term retention, and that learners assume OBEs are easier than CBEs and may consequently study less for an OBE (Durning et al. 2016). OBE advocates argue that OBEs provide the opportunity to test high-order thinking, reflective thinking and problem-solving skills, beyond root learning.
OBEs are more authentic to clinical practice. Online OBEs can provide a simulation of the point-of-care resources that physicians are expected to use in their clinical practice. Views on what defines a competent health care professional are changing. Where formerly the focus lay almost entirely on the possession of knowledge, currently physicians are expected to effectively use external point-of-care knowledge (Durning et al. 2016). They may also heighten learner awareness of the importance of aptly identifying and utilizing best available evidence in clinical decision-making (Zagury-Orly et al. 2021).
Durning et al reviewed the research to date on open-book and closed-book exams and concluded that:
Exam preparation. Research exploring exam preparation was equivocal with respect to whether students prepare differently (or at greater length) for CBEs or OBEs. When differences did exist, they tended to show that participants studied more when they expected a CBE.
Test anxiety. Evidence suggests that research exploring exam preparation was equivocal with respect to whether students prepare differently (or at greater length) for CBEs or OBEs. When differences did exist, they tended to show that participants studied more when they expected a CBE.
Exam performance. Studies comparing exam performance appear to favor CBEs. However, the combination of relatively little experience with OBEs and the differences in exam preparation noted in several investigations highlighted in this section leave open the possibility that OBE performance could be improved through instructing students about OBEs or providing practice tests.
Psychometrics and logistics: studies that compared CBEs with OBEs with respect to their influence on the time required to take the test found that students took 10% to 60% longer to complete OBEs.10,30,32 Thus, if one controls for amount of testing time, it is likely that fewer questions would be asked in OBE format, and, hence, the reliability of the equivalent CBE-formatted exam can be anticipated to be higher.
Testing effects. Both OBE and CBE demonstrate testing effects that are roughly equivalent.
(Durning et al. 2016)
There does not appear to be sufficient evidence for relying solely on OBE or CBE formats (Durning et al. 2016), but open-book can effectively complement closed-book exams as a blended approach (Zagury-Orly et al. 2021). Open-book exams might be justified to test students on the lowest level of Bloom’s taxonomy (remember) but may not be appropriate on the higher levels (understanding, applying, analyzing, evaluating and creating).
Workplace-based assessment
Workplace-based assessment (WBA) plays a crucial role in capturing feedback and consolidating learning in various professional settings. Whilst ‘bedside’ WBA formats (e.g. mini CEX) became challenging during the pandemic, collecting evidence of learning was still vital to continue to monitor student development and provide support.
E-portfolios with built-in WBA forms have provided a useful aid to collate evidence of learning, allowing students to compile a comprehensive record of their experiences, reflections and achievements, providing a holistic view of the student’s development over time. E-portfolios with learning analytics can increase the quality and efficiency of workplace-based feedback and assessment in professional education (van der Schaaf et al. 2017). E-portfolios are increasingly used to assess workplace-based learning. Much of this advancement has been facilitated by the use of mobile devices to capture workplace-based assessment data.
Challenges
Technology enabled institutions to rapidly transform from face-to-face to online education. Looking beyond the Covid-19 pandemic, technology opens up promising avenues for innovating assessment practices. Nevertheless, incorporating technology into assessment and more widely into education brings significant challenges.
The introduction of Technology-Enabled Assessment (TEA) raises a crucial question about how to instigate technological changes that promote digital equity, a complex challenge classified as a “wicked challenge” according to the EDUCAUSE framework.
“Digital equity refers to comparable access to technology, particularly to broadband connectivity sufficient to access unbiased, uncensored content and to enable full participation on the World Wide Web.” (Alexander B et al. 2019).
Digital divide
Certain institutions have faced challenges in adopting teaching, learning, and assessment practices that necessitate the use of technology. Furthermore, disparities exist as not all students have access to the latest technology. Global disparities persist in broadband access, exhibiting inequalities across various factors including income, education, gender, age, ability status, native language, as well as national, regional, and cultural dimensions.
Aristovnik et al.’s study reveals (with a sample of 30,383 higher education students from 62 countries) that the Covid-19 crisis had a stronger negative effect on students with a lower living standard and students from less developed regions (in Africa and Asia).
“On all three continents [South America, Asia, and Africa], the key challenges are problems with an underdeveloped Internet network, lack of and inexperience in using ICT equipment, and the fact that the only available devices for participating in online classes are mobile phones.
The geographical differences in the results are alarming, especially for Africa and Asia [15,45,64,83] and should be understood as an important signal for international, national, and higher education authorities to ensure they appropriately respond with adequate policy recommendations concerning different aspects of student life in order to minimise the gap in students’ opportunities among different parts of the world.” (Aristovnik et al. 2020).
Achievement gap (difficulty challenge)
Educational institutions have increasingly integrated technology-enhanced assessment and adaptive courseware to offer students immediate feedback and better access to content designed to help them master course material. However, despite these advancements, achieving degree completion in higher education faces obstacles that extend beyond these efforts, and closing the achievement gap continues to be a difficult challenge. There is also a risk that the introduction of TEA will widen the already large attainment gaps, between different groups of learners in an institution or between/across institutions (Rahim 2020; Hegazy et al. 2021).
One critical factor is the uneven access to high-speed networks and hardware, especially beyond personal mobile phones. The availability of larger and more powerful devices remains disparate, even in affluent nations, highlighting the challenges associated with hardware access in education (Bryan Alexander et al).
UNESCO noted that there are two dimensions to digital accessibility: technical (infrastructure for connectivity and capacity of devices) and socio-economic (affordability and students’ digital capabilities). Unequal access to technologies and subsequently to TEA is linked to and can exacerbate, existing inequalities of income and resources between and within communities (UNESCO 2018).
There is a continued need to avoid embracing the myth that the next generation of learners are so-called ‘digital natives’ who easily embrace the use of new technology in learning and assessment because they use mobile devices fluently in their personal lives (Tapscott 1998; Prensky 2001; Bennett et al. 2008; Jones 2010; Jones et al. 2010). Whilst learners and faculty have increased technology immersion, this primarily exists in their social/personal spaces; gaps between their use of digital technology in their daily life and in learning have been reported in previous studies (Selwyn 2010; Pyor € €al€a et al. 2019). HPE therefore still needs to focus on the support that all learners and educators will need, to make TEA effective.
Conclusion
The pandemic crisis led to creative responses and saw a rapid growth of innovation in technology use in education, including technological solutions to assessment (Fuller et al. 2020). For this to be really impactful, we need diversity and dissemination across the globe. Issues of affordability, digital equity and equitable access to education will be the driving forces that must shape our strategic approaches to TEA.
References
Albinson, Paul & Cetinkaya, Deniz & Orman, Tim. (2020). Using Technology to Enhance Assessment and Feedback: A Framework for Evaluating Tools and Applications. 10.1145/3383923.3383940.
Alexander B, Ashford-Rowe K, Barajas-Murphy N, Dobbin G, Knott J, McCormack M, Pomerantz J, Seilhamer R, Weber N. 2019. EDUCAUSE horizon report: 2019 higher education edition. Louisville: EDUCAUSE; [accessed 2022 Jan 21]. https://library.educause.edu/-/media/files/ library/2019/4/2019horizonreport.pdf
Amin, Zubair & Boulet, John & Cook, David & Ellaway, Rachel & Fahal, Ahmed & Maley, Moira & Østergaard, Doris & Ponnamperuma, Gominda & Wearn, Andy & Ziv, Amitai. (2011). Technology-enabled assessment of health professions education: Consensus statement and recommendations from the Ottawa 2010 conference. Medical teacher. 33. 364-9.
Aristovnik A, Kerzic D, Ravselj D, Tomazevic N, Umek L. (2020). Impacts of the COVID-19 pandemic of life of higher education students: a global perspective. Sustainability. 12(20):8438.
Bala L, van der Vleuten C, Freeman A, Torre D, Heeneman S, Sam AH. COVID-19 and programmatic assessment. Clin Teach. 2020 Aug;17(4):420-422.
Bengtsson, Lars. (2019). Take-Home Exams in Higher Education: A Systematic Review. Education Sciences. 9. 267.
Deeley, Susan. (2017). Using technology to facilitate effective assessment for learning and feedback in higher education. Assessment & Evaluation in Higher Education. 43. 10.1080/02602938.2017.1356906.
Durning SJ, Dong T, Ratcliffe T, Schuwirth L, Artino AR, Boulet JR, Eva K. (2016). Comparing Open-Book and Closed-Book Examinations: A Systematic Review. Academic Medicine. 91(4):583–599.
Fuller, Richard & Joynes, Viktoria & Cooper, Jon & Boursicot, Katharine & Roberts, Trudie. (2020). Could COVID-19 be our ‘There is no alternative’ (TINA) opportunity to enhance assessment?. Medical Teacher. 42. 1-6.
Fuller, Richard & Goddard, Viktoria & Nadarajah, Vishna & Treasure-Jones, Tamsin & Yeates, Peter & Scott, Karen & Webb, Alexandra & Valter, Krisztina & Pyörälä, Eeva. (2022). Technology enhanced assessment: Ottawa consensus statement and recommendations. Medical Teacher. 44. 1-15.
Grover S, Pandya M, Ranasinghe C, Ramji SP, Bola H, Raj S. Assessing the utility of virtual OSCE sessions as an educational tool: a national pilot study. BMC Med Educ. 2022 Mar 15;22(1):178.
Hara, S., Ohta, K., Aono, D. et al. (2023). Feasibility and reliability of the pandemic-adapted online-onsite hybrid graduation OSCE in Japan. Adv in Health Sci Educ.
Hytönen , H , Näpänkangas , R , Karaharju-Suvanto , T , Eväsoja , T , Kallio , A , Kokkari , A , Tuononen , T & Lahti , S. (2021). ‘ Modification of national OSCE due to COVID-19-Implementation and students’ feedback ‘ , European Journal of Dental Education , vol. 25 , no. 4 , pp. 679-688.
Kakadia, Rahen & Chen, Emily & Ohyama, Hiroe. (2020). Implementing an online OSCE during the COVID-19 pandemic. Journal of Dental Education. 85. 10.1002/jdd.12323.
Mooney JS, Cappelli T, Byrne-Davis L, Lumsden CJ. (2014). How we developed eForms: an electronic form and data capture tool to support assessment in mobile medical education. Med Teach. 2014 Dec;36(12):1032-7.
Ricketts, Chris & Bligh, John. (2011). Developing a “Frequent Look and Rapid Remediation” Assessment System for a New Medical School. Academic medicine : journal of the Association of American Medical Colleges. 86. 67-71.
Roman P, Ruiz-Gonzalez C, Rodriguez-Arrastia M, Granero-Molina J, Fernandez-Sola C, Hernandez-Padilla JM. (2022). A serious game for online-based objective structured clinical examination in nursing: a qualitative study. Nurse Educ Today. 109:105246.
Ryan, A., O’Mara, D., & Tweed, M. (2023). Evolution or revolution to programmatic assessment: Considering unintended consequences of assessment change. Focus on Health Professional Education: A Multi-Professional Journal 2023, 24(2), 185–195.
Sam, Amir & Field, Samantha & Collares, Carlos & Van der Vleuten, Cees & Wass, Val & Melville, Colin & Harris, Joanne & Meeran, Karim. (2018). Very-short-answer questions: reliability, discrimination and acceptability. Medical Education. 52. 10.1111/medu.13504.
Sambell K, McDowell L, Montgomery C. (2012). Assessment for learning in higher education. Oxford (UK): Routledge. Sierles F, Hendrickx I, Circle S. 1980.
Schuwirth L, van der Vleuten C, Durning SJ. What programmatic assessment in medical education can learn from healthcare. Perspect Med Educ 2017;6(4):211–215.
Van der Schaaf M, Donkers J, Slof B, Moonen-van Loon J, van Tartwijk J, Driessen E, Badii A, Serban O, en Cate O. (2017). Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics. Educ Tech Res Dev. 65(2):359–380.
Van der Vleuten, Cees. (2016). Revisiting ‘Assessing professional competence: from methods to programmes’. Medical Education. 50. 885-888. 10.1111/medu.12632.
Van der Vleuten, Cees & Schuwirth, Lambert. (2005). Assessing professional competence: From methods to programmes. Medical education. 39. 309-17.
Van der Vleuten, Cees & Schuwirth, L & Driessen, E & Dijkstra, Joost & Tigelaar, Dineke & Baartman, L.K.J. & Tartwijk, Jan. (2012). A model for programmatic assessment fit for purpose. Medical teacher. 34. 205-14. 10.3109/0142159X.2012.652239.
Zagury-Orly, Ivry & Durning, Steven. (2020). Assessing open-book examination in medical education: The time is now. Medical Teacher. 43.
All Things Education 