Randomised control trial background briefing paper

Can’t publish the full thing but the cost reduction/reallocations based on previous baseline I created at MHRA a few years ago and use occasionally when needed.

pdf version & below

Consultants Mandatory Training replaced by online learning

We are proposing to replace the face to face training sessions with a full online learning programme, following feedback from consultants during the next 2/3 months. Consultants Mandatory Training is currently delivered as a series of face to face training sessions during a day. The costs for the Trust include the consultant clinical time away from patient care and trainer time for preparation and delivery. The Trust has recently implemented a new learning management system – the Learning Hub which allows greater ease of access to a wider range of online learning including mandatory training modules.

The Learning & Development team have been working with subject matter experts overseen by the PGME Director to develop online learning content suitable for consultants. This means that they can complete the modules during their own time and location because the hub is hosted securely on the internet with a design that allows mobile and computer access.

As humans, consultants face increased information overload and retention issues. Face to face training may address an organisational need at the time but it takes them away from patient care and is not known how they need to use or apply any new knowledge or behaviours back in their work environment. Sometimes there may be a delay in needing to apply any new knowledge or behaviours which mean that retention from the face-face training session will continue to decrease.
The availability of online packages and resources on the hub helps address information retention and accessibility of information as needed including use of a mobile device. It is hoped this evaluation exercise will provide evidence about the effectiveness of learning interventions to meet organisational risks or needs as well as at a personal level.

Feedback indicates that consultants do not wish to spend time in classroom training for mandatory subjects. This analysis identifies areas where time and cost can be saved or reallocated by using eLearning. A consultant will spend approximately ½ day of clinical time attending mandatory training. Completing eLearning packages and/or refresher activities online can be done either outside of clinical time if preferred or approximately ¼ day continuously using the hub.

Baseline Cost Reductions / Reallocations:
Approximate Trainer & Training hours for Consultants Total days
2016 Total Days spent in Mandatory Training (total attendee hrs) x days (x hrs)
2016 Total Days Trainer & Support x days (x hrs)
2016 total days consultation, design, evaluation, admin x days
2016 total days delivery x days
2016 Hours spent in L&D admin support x days
2016 Total attendee, trainer & admin days x days
2016 Days spent in Consultant MT x days
2016 Days Trainer & Support for Consultants MT x days
Avg total trainer & admin Cost £x
Avg total attendee Cost £x
Avg Total Locum Cover Cost £x
Avg Total Cost £x

Evaluation Design
Whilst there is a suggested cost baseline, we do not have a baseline for what is an effective learning intervention as opposed to other interventions to meet the needs of mandatory compliance.

The Trust currently use feedback questionnaires similar to level 1 of the Kirkpatrick model which attempt to gauge effectiveness based on a reaction / satisfaction after the learning intervention1. Additional levels are learning – the extent to which participants change attitudes, improve knowledge or increase skills, behaviour where the new knowledge, skills or attitudes transfer into their job (e.g. through an assessment) or a further course and finally organisational results from the change in learners.2

Critiques of the model suggest that it removes the focus away from instructional design (a core component of online learning packages development), it does not accurately measure whether learners remember anything or whether they are motivated to apply what they’ve learned. This could be true in an online or face-face context.

Other critiques are that it falsely correlates feedback questionnaires with learning results3, or causal links between positive reactions and additional levels assuming that each level of evaluation provides data that is more informative than the last 4, insufficient evidence about how evaluation can inform whether good or bad behaviours which could then affect organisational risks can occur.

So evaluation of mandatory training for consultants could inaccurately identify whether this learning intervention has helped – such as a reduction in HAIs where consultants sit in a face-face session or online learning package to see examples of how to wash hands. You could also bring in clinical case studies where consultants could evaluate the evidence of hand washing in different clinical scenarios but this is also not possible to evaluate whether a more interesting learning intervention in a face-face session or online is effective in meeting a reduction in HAIs.

Effective learning evaluations should help organisations and individuals apply the principle of beneficence – an ethical duty where “actions must be taken when the opportunity arises to actively contribute to the health and welfare of clients and stakeholders”5

There are risks that a Kirkpatrick model will not be able to convey accurate information about whether risks have been addressed to stop future harm or loss6
Applying the principle of beneficence, as learning professionals we are ethically obligated to provide evidence of whether evaluation of a learning intervention is effective. That should extend beyond the organisation so that this exercise can be evaluated by a wider audience.

Alternative evaluation models suggest that a mix of accumulation of evidence over time and qualitative studies would be useful. Whilst is is not possible to have a group of consultants with a placebo (no mandatory training) currently, it is hoped this exercise could inform discussion to see if this is possible for future learning interventions. If we don’t then we are failing in our ethical duty.
Method
A group of consultants who have registered to attend upcoming mandatory training sessions will be randomly selected to complete the training using online learning through the hub. An additional consultant will act as observer during the face to face training sessions.
Qualitative data will be gathered through short interviews and anecdotal snippets provided by..

References
1. Kirkpatrick, D. L. (2001). Evaluating Training Programs: The Four Levels ,2 ed. San Francisco, CA: Berrett-Koehler
2. Owston, R (2007), Models and Methods for Evaluation, Journal for the Theory of Social Behaviour 1(1):75 – 93 DOI: 10.1111/j.1468-5914.1971.tb00166.x
3. Thalheimer W(2015) Kirkpatrick Model Good or Bad, http://www.willatworklearning.com/2015/03/kirkpatrick-model-good-or-bad-the-epic-mega-battle.html, https://youtu.be/QucqCxM2qW4
4. Bates R, (2004), A critical analysis of evaluation practice: the Kirkpatrick model
and the principle of beneficence, Evaluation and Program Planning 27 , 341-7 https://aetcnec.ucsf.edu/sites/aetcnec.ucsf.edu/files/A%20critical%20analysis%20of%20evaluation%20practice_0.pdf
5. Bates R, (2004), A critical analysis of evaluation practice: the Kirkpatrick model
and the principle of beneficence, Evaluation and Program Planning 27 , 341-7 https://aetcnec.ucsf.edu/sites/aetcnec.ucsf.edu/files/A%20critical%20analysis%20of%20evaluation%20practice_0.pdf
6. Bates R, (2004), A critical analysis of evaluation practice: the Kirkpatrick model
and the principle of beneficence, Evaluation and Program Planning 27 , 341-7
https://aetcnec.ucsf.edu/sites/aetcnec.ucsf.edu/files/A%20critical%20analysis%20of%20evaluation%20practice_0.pdf
7. Owston, R (2007), Models and Methods for Evaluation, Journal for the Theory of Social Behaviour 1(1):75 – 93 DOI: 10.1111/j.1468-5914.1971.tb00166.x

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s