#bruneldigitalexams – digital assessment – updated

Presentations now available online


Will update & republish this post throughout today

Brunel (use Blackboard) had HEFCE grant for evaluating pilot – using Uniwise (has open API, can integrate with Moodle):

https://t.co/UzNoMdNBRv

In Scandinavia, exams are any type of summative assessment, in UK, generally means timed exams.

Dr Simon Kent, Brunel – digital exams

https://www.brunel.ac.uk/about/education-innovation/Digital-Assessment-Brunel/About-Digital-Assessment-Brunel

Previously low engagement with electronic submission/exams. Pilot in Engineering & Physical Sciences, up to 3k students.UG & PG. How to scale – tech problems easier to fix than organisational ones. HEFCE grant received during scaling. 18 BYOD exams early 2018, going to 25 in a couple of weeks time.

students don’t want to prepare, staff don’t want too much change but like exams,  admin staff positive until system goes live but like post exam processing

Broadly going successfully – lessons/issues include power, wifi,laptop loans, tech support.

Increased political pressure – less resources, want high standards without additional funding. Initiative overload for staff & students, digital assessment not just another thing to do.

Creating a new electronic management of assessment workflow:

http://pbs.twimg.com/media/Dbsz1y7WAAAdkeb.jpg

https://twitter.com/fionajharvey/status/989441551460982784

Dr Torben K Jensen, Aarhus university

21 employees in Centre for Teaching & Learning. Blackboard. Use wiseflow (Uniwise) and Blue for evaluation. 750 flows / written exams per semester. Excuse crappy photo – their evaluation so far:

Research philosophy – students work as researchers – exercises, projects, supervision, feedback.
MCQ rare in Denmark

By 2023 –  uni cautious strategy – 20% in class, 57% TEL, 20% blended, 3% fully online

Using same learning objects/ activities in learning & assessment – including multimedia

Am asking around today to see if anyone using animations in life sciences assessment – just curious at this point re practicality, possibly Norway at the moment

If new generations never writing by hand, is it useful to assess this way,  couple of presentations later on this.

Aarhus have lowered economic and admin costs,  saved over DKK 1m per year.

Dr Liz Masterman, Oxford http://digitaleducation.ox.ac.uk

Handwritten vs typed exams equivalence, incl

https://projects.it.ox.ac.uk/edu-it-e-exams

Difference? Did literature review 2000 onwards, 35 articles – themes:

Student impact & anxiety about change of tool, tech failure, writing fluency, cognitive processing

Typing proficiency greater effext than computer experience

Staff perceptions – legibility errors more visible in typed scripts, empathy re poor handwriting; typed answers shorter; marking – studies 10 yrs old – skim reading, navigation, public/private annotations, no significant diff in severity, accuracy.

Higher marks typed vs handwritten not signicantly different.

Keystroke tracking, useful for evaluation but data protection.

Trial at Oxford last week, Inspera – 4 exams, 70 UG & taught masters students. Formative – Theology, computing, Mock exams – law, finance, some BYOD, some uni supplied, all typed exams. Evaluation in progress.

Brunel poster

Athina Chatzigavrill, Geraldine Foley, LSE

Handwritten vs typed – student perceptions

3 pilots typed exams – formative essays:

2 with law – timed take-home using Examsoft, 1 with government – timed, on campus using BYOD

Found students mixed views of typing vs handwriting, tech support & training. Logistical issues e.g. wifi for BYOD ok for pilot but not necessarily scaling up

Writing speed & editing – bad handwriting like typed, some found easier to edit e.g. deleting, structuring but felt it involves more critical analysis when handwriting

Concerns about typing / eyes on screen continuously e.g. 2 hr exam. Concerns that examiners expect more if typed, want more practice, feedback.

Harder to find quiet space off campus. Temperature of exam rooms an issue and noise from simultaneous keyboard typing. Concern about power with BYOD, if laptop dies during exam…

Implementation considerations:

Each student needs power socket. Institutional wifi, data protection and security, students who can’t bring suitable device, international students with different devices, software.

Next step – law deot doing trial with Moodle submission,  Digiexam ipads & keyboards doing summative assessment this summer.

eProctoring – assessment of remote exam completion

Dr Rebecca Gill, Newcastle – expanding secure, online provision, now doing 16-20,000 students, 150 approx high stakes exams, running from pilot in 2010.

https://www.ncl.ac.uk/ltds/about/staff/profile/rebecca.gill#tab_profile

Use Blackboard and Wiseflow, piloting BYOD next month

Lock down using Respondus.

Have printed paper back up version of exam.

Constraints –

staff capacity,

physical space & timetabling – 7 venues, computing clusters, pressure on spaces that students use for revision,study

Ethics

Tech – formulas, equations in Blackboard, version control, so have handwritten section of exams

Use optical mark reader – Speedwell, Numbas maths testing open source http://numbas.org.uk

Now looking at feasibility of BYOD – 85% say have Windows/Mac they can use. Students want technical check beforehand.

Doing full evaluation after pilot.

Dr Richard Walker, York – high stakes testing challenges

https://www.slideshare.net/mobile/RichardM_Walker/establishing-high-stakes-computerbased-testing-through-blackboard-as-a-supported-service-an-institutional-perspective-on-challenges-and-lessons-learned

https://elearningyork.wordpress.com/about/richard-walker/

Lessons learnt:

Network failure, timetabling failure, server connectivity, latency, invigilators unfamiliar with exam protocol when hired on sessional basis

Manage risk – protocols and technology

Mitigate – exam design, support design & communication. Have mirrored VLE but stripped down to assessment essentials, lockdown browser, restrict access to server incl IP, exam duration/timing, specific accounts not normal student credentials. System restart prior to exam, priming caches – review exam & content. Randomisation, consideration of inout e.g. short & long answer vs mcq – hit on servers.

Put questions in blocks for weighting, difficulty. suppirt staff in question development & system training. Geoffrey Crisp, Australia https://draaijeronassessmentandtesting.wordpress.com/2012/02/25/e-assessment-handbook-geoffrey-crisp-where-have-you-been-all-my-life/

video tutorial for students, mature students esp disadvantage.

http://leaplearningframework.org/

http://www.researchinlearningtechnology.net/index.php/rlt/article/view/30083

Alice La Rooy, Claudia Cox, Brunel

Pre exam support – email notifications and team visits stydents in final lecture before exams. Hold drop-ins for students. User help section including interactive quiz.

Wifi boosters in exam rooms, power supply for 30% of rooms max capacity

Non BYOD notify using a device declaration form, approx 20% students need. Simple startup guidelines on each desk, also have student associate learning technologists on site (sessional basis funding)

19% of students need power recharge during exam. Use Wiseflow to monitor effectiveness of exam e.g. track live student non activity.

Students who didn’t attend drop-in session expected people to set up their device for them.

Use of radio communication across different local venues

Suggestion that SALTs wear coloured Tshirts so easy to identify.

Gunhild Raunsgard, Marita Kristiansen – West Norway Uni, Norwegian School Economics

Digitisation of exams for student exams. Pilot 2014-2016, 3 FT admins.

National competitive tender process for 20 unis/colleges. UNINETT. Common student admin system across Norway.

Spring 2018: 43-55% digital written exams, 100% digital take home exams. Use Wiseflow.

Mix of BYOD. Now including maths, engineering, sciences due to dusplay of equations, formulas, images, calculations etc

All students have power sockets, emergency computers, easir with larger exam rooms. BYOD easier because can require by law that students bring their own device but can’t make requirement for international / exchange students.

Next step – automating appeals.


Mia Morkeby Johnson
, Uni college Copenhagen

20,000 digital exams in Wiseflow 2017

Started with midwifery, now biomedical science live in January 2018. Met with all academic staff, have to use by 2019 – when & how up to them – staff submitted details, preferred dates then planned centrally. Using BYOD.

Did demo sessions with 50 students, lecturer and TEL team. Students tried lockdown browser before these demos. Qs incl what happens if lose wifi etc

Invigilators arrive an hour in advance of exam and do training.

On exam day, open/startup – countdown clock to start of exam then enter password.

3% had system requirement issues on day


Bianca Dryer Hyre
, Uni college Lillbaelt, Denmark

In Denmark, no handwritten exams.

Efficiency savings. Driven by national finance law change and directive from Ministry of Education and Research. Increased demand for e assessment.

Savings increase in assignment submission, staff not a reduction but freed up academic time, some tasks handled by admins, GDPR friendly, better task management if someone off sick, better communication between students and staff, better quality, greater transparency.

Would like easier remuneration for assessors

Other comments from today

Noise of keyboards affecting others. One person going to do audio monitoring test for exam in early May.

Impero software mentioned. We have done some early testing in our team only with Safeexam Browser: https://safeexambrowser.org/download_en.html

Re animations, no one currently using 3D animations in assessment questions but keeping an eye on authoring. Can do animated gifs (e.g.molecular binding for biomed) in Wiseflow. In Denmark, tried using chemical keyboard but no animations as yet. We discussed possibility of filming closeup of lecturer drawing during a lecture and using as video rather than asking them to learn an animation tool. They would also have a memory trigger if they attended or lecture captured – we don’t have video capture at the moment just audio & slides and not all lectures http://roehamptonlearning.com/eLearningServices/help-guides/recap.

Mixed exams with drawing section which could be scanned and handed in , Wiseflow can do but it is another manual task so takes longer or use webcam on exam computer to scan drawing.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

%d bloggers like this: