Identify indicators of success

Right Decision Service implementation guide
National

How:

This guide encourages you to use the four-stage evaluation framework developed for the Right Decision Service, summarised in figure 3 below. This is based on a theory of change which describes the relationship between different levels of impact of decision support.

Each stage in this chain is dependent on the previous stage. The impact of external confounding factors such as competing initiatives and work pressures becomes stronger at each successive stage, and you should manage the barriers and facilitators identified in stage 1.5 above to maximise success at each stage.

At the initial levels of the chain, it is usually quite straightforward to relate your success measures to the influence of decision support. However, as you move along the chain, you need to give more thought to how you demonstrate that improvement is related (at least in part) to the impact of the decision support tools.

 

Figure 3: The four-stage evaluation framework for the RDS

Figure 3: the four stage evaluation framework for the RDS

 

 

Table 2: Some examples of indicators or measures of success for each stage in the evaluation framework

Outcomes

Examples of indicators of success

Stage 1: Stakeholder engagement and valuing of the service (“Reach and reaction”)

Usage statistics for the decision support tool.

User feedback – can be gathered informally or more formally through questionnaire and interview.

Stage 2: Development of new knowledge, skills, and attitudes

Numbers undertaking training on new decision support tool.

User feedback on how knowledge and skills in a particular area of practice has improved as a result of using decision support.

Creation of new decision support roles for knowledge manager and practitioners.

Stakeholder feedback on improved understanding of benefits of decision support.

Stage 3: Change in practice, policies and behaviours

 

Data on changes in prescribing, test ordering, referral and triage, with supporting evidence that decision support has contributed to this change.

Case studies of changes in service model as a result of implementing decision support – e.g. arrangements for referral for shared decision-making discussions or medicines review following decision support recommendations.

Stage 4: Improved outcomes for systems, people, and communities.

Reduction in costs – e.g. of prescribing, test ordering, treatment of complications.

Reduction in emergency admissions.

Reduction in adverse events.

At this outcome level it is important to gather evidence that the change can be attributed - at least in part - to decision support. This may be achieved, for example,  through stakeholder interviews, focus groups and case studies.