Uncategorized

#bruneldigitalexams – digital assessment – updated

Presentations now available online


Will update & republish this post throughout today
Brunel (use Blackboard) had HEFCE grant for evaluating pilot – using Uniwise (has open API, can integrate with Moodle):
https://t.co/UzNoMdNBRv
In Scandinavia, exams are any type of summative assessment, in UK, generally means timed exams.
Dr Simon Kent, Brunel – digital exams
https://www.brunel.ac.uk/about/education-innovation/Digital-Assessment-Brunel/About-Digital-Assessment-Brunel
Previously low engagement with electronic submission/exams. Pilot in Engineering & Physical Sciences, up to 3k students.UG & PG. How to scale – tech problems easier to fix than organisational ones. HEFCE grant received during scaling. 18 BYOD exams early 2018, going to 25 in a couple of weeks time.
students don’t want to prepare, staff don’t want too much change but like exams,  admin staff positive until system goes live but like post exam processing
Broadly going successfully – lessons/issues include power, wifi,laptop loans, tech support.
Increased political pressure – less resources, want high standards without additional funding. Initiative overload for staff & students, digital assessment not just another thing to do.
Creating a new electronic management of assessment workflow:
http://pbs.twimg.com/media/Dbsz1y7WAAAdkeb.jpg
https://twitter.com/fionajharvey/status/989441551460982784
Dr Torben K Jensen, Aarhus university
21 employees in Centre for Teaching & Learning. Blackboard. Use wiseflow (Uniwise) and Blue for evaluation. 750 flows / written exams per semester. Excuse crappy photo – their evaluation so far:
Research philosophy – students work as researchers – exercises, projects, supervision, feedback.
MCQ rare in Denmark
By 2023 –  uni cautious strategy – 20% in class, 57% TEL, 20% blended, 3% fully online
Using same learning objects/ activities in learning & assessment – including multimedia
Am asking around today to see if anyone using animations in life sciences assessment – just curious at this point re practicality, possibly Norway at the moment
If new generations never writing by hand, is it useful to assess this way,  couple of presentations later on this.
Aarhus have lowered economic and admin costs,  saved over DKK 1m per year.
Dr Liz Masterman, Oxford http://digitaleducation.ox.ac.uk
Handwritten vs typed exams equivalence, incl
https://projects.it.ox.ac.uk/edu-it-e-exams
Difference? Did literature review 2000 onwards, 35 articles – themes:
Student impact & anxiety about change of tool, tech failure, writing fluency, cognitive processing
Typing proficiency greater effext than computer experience
Staff perceptions – legibility errors more visible in typed scripts, empathy re poor handwriting; typed answers shorter; marking – studies 10 yrs old – skim reading, navigation, public/private annotations, no significant diff in severity, accuracy.
Higher marks typed vs handwritten not signicantly different.
Keystroke tracking, useful for evaluation but data protection.
Trial at Oxford last week, Inspera – 4 exams, 70 UG & taught masters students. Formative – Theology, computing, Mock exams – law, finance, some BYOD, some uni supplied, all typed exams. Evaluation in progress.
Brunel poster

Athina Chatzigavrill, Geraldine Foley, LSE
Handwritten vs typed – student perceptions
3 pilots typed exams – formative essays:
2 with law – timed take-home using Examsoft, 1 with government – timed, on campus using BYOD
Found students mixed views of typing vs handwriting, tech support & training. Logistical issues e.g. wifi for BYOD ok for pilot but not necessarily scaling up
Writing speed & editing – bad handwriting like typed, some found easier to edit e.g. deleting, structuring but felt it involves more critical analysis when handwriting
Concerns about typing / eyes on screen continuously e.g. 2 hr exam. Concerns that examiners expect more if typed, want more practice, feedback.
Harder to find quiet space off campus. Temperature of exam rooms an issue and noise from simultaneous keyboard typing. Concern about power with BYOD, if laptop dies during exam…
Implementation considerations:
Each student needs power socket. Institutional wifi, data protection and security, students who can’t bring suitable device, international students with different devices, software.
Next step – law deot doing trial with Moodle submission,  Digiexam ipads & keyboards doing summative assessment this summer.
eProctoring – assessment of remote exam completion
Dr Rebecca Gill, Newcastle – expanding secure, online provision, now doing 16-20,000 students, 150 approx high stakes exams, running from pilot in 2010.
https://www.ncl.ac.uk/ltds/about/staff/profile/rebecca.gill#tab_profile
Use Blackboard and Wiseflow, piloting BYOD next month
Lock down using Respondus.
Have printed paper back up version of exam.
Constraints –
staff capacity,
physical space & timetabling – 7 venues, computing clusters, pressure on spaces that students use for revision,study
Ethics
Tech – formulas, equations in Blackboard, version control, so have handwritten section of exams
Use optical mark reader – Speedwell, Numbas maths testing open source http://numbas.org.uk
Now looking at feasibility of BYOD – 85% say have Windows/Mac they can use. Students want technical check beforehand.
Doing full evaluation after pilot.
Dr Richard Walker, York – high stakes testing challenges
https://www.slideshare.net/mobile/RichardM_Walker/establishing-high-stakes-computerbased-testing-through-blackboard-as-a-supported-service-an-institutional-perspective-on-challenges-and-lessons-learned
https://elearningyork.wordpress.com/about/richard-walker/
Lessons learnt:
Network failure, timetabling failure, server connectivity, latency, invigilators unfamiliar with exam protocol when hired on sessional basis
Manage risk – protocols and technology
Mitigate – exam design, support design & communication. Have mirrored VLE but stripped down to assessment essentials, lockdown browser, restrict access to server incl IP, exam duration/timing, specific accounts not normal student credentials. System restart prior to exam, priming caches – review exam & content. Randomisation, consideration of inout e.g. short & long answer vs mcq – hit on servers.
Put questions in blocks for weighting, difficulty. suppirt staff in question development & system training. Geoffrey Crisp, Australia https://draaijeronassessmentandtesting.wordpress.com/2012/02/25/e-assessment-handbook-geoffrey-crisp-where-have-you-been-all-my-life/
video tutorial for students, mature students esp disadvantage.
http://leaplearningframework.org/
http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083
Alice La Rooy, Claudia Cox, Brunel
Pre exam support – email notifications and team visits stydents in final lecture before exams. Hold drop-ins for students. User help section including interactive quiz.
Wifi boosters in exam rooms, power supply for 30% of rooms max capacity
Non BYOD notify using a device declaration form, approx 20% students need. Simple startup guidelines on each desk, also have student associate learning technologists on site (sessional basis funding)
19% of students need power recharge during exam. Use Wiseflow to monitor effectiveness of exam e.g. track live student non activity.
Students who didn’t attend drop-in session expected people to set up their device for them.
Use of radio communication across different local venues
Suggestion that SALTs wear coloured Tshirts so easy to identify.
Gunhild Raunsgard, Marita Kristiansen – West Norway Uni, Norwegian School Economics
Digitisation of exams for student exams. Pilot 2014-2016, 3 FT admins.
National competitive tender process for 20 unis/colleges. UNINETT. Common student admin system across Norway.
Spring 2018: 43-55% digital written exams, 100% digital take home exams. Use Wiseflow.
Mix of BYOD. Now including maths, engineering, sciences due to dusplay of equations, formulas, images, calculations etc
All students have power sockets, emergency computers, easir with larger exam rooms. BYOD easier because can require by law that students bring their own device but can’t make requirement for international / exchange students.
Next step – automating appeals.

Mia Morkeby Johnson
, Uni college Copenhagen
20,000 digital exams in Wiseflow 2017

Started with midwifery, now biomedical science live in January 2018. Met with all academic staff, have to use by 2019 – when & how up to them – staff submitted details, preferred dates then planned centrally. Using BYOD.
Did demo sessions with 50 students, lecturer and TEL team. Students tried lockdown browser before these demos. Qs incl what happens if lose wifi etc
Invigilators arrive an hour in advance of exam and do training.
On exam day, open/startup – countdown clock to start of exam then enter password.
3% had system requirement issues on day

Bianca Dryer Hyre
, Uni college Lillbaelt, Denmark
In Denmark, no handwritten exams.
Efficiency savings. Driven by national finance law change and directive from Ministry of Education and Research. Increased demand for e assessment.
Savings increase in assignment submission, staff not a reduction but freed up academic time, some tasks handled by admins, GDPR friendly, better task management if someone off sick, better communication between students and staff, better quality, greater transparency.
Would like easier remuneration for assessors

Other comments from today

Noise of keyboards affecting others. One person going to do audio monitoring test for exam in early May.
Impero software mentioned. We have done some early testing in our team only with Safeexam Browser: https://safeexambrowser.org/download_en.html
Re animations, no one currently using 3D animations in assessment questions but keeping an eye on authoring. Can do animated gifs (e.g.molecular binding for biomed) in Wiseflow. In Denmark, tried using chemical keyboard but no animations as yet. We discussed possibility of filming closeup of lecturer drawing during a lecture and using as video rather than asking them to learn an animation tool. They would also have a memory trigger if they attended or lecture captured – we don’t have video capture at the moment just audio & slides and not all lectures http://roehamptonlearning.com/eLearningServices/help-guides/recap.
Mixed exams with drawing section which could be scanned and handed in , Wiseflow can do but it is another manual task so takes longer or use webcam on exam computer to scan drawing.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.