Selecting Surveys to Measure Treatment Impact

Laura Mills, Ph.D. (QM, Psych) Director, Research & Evaluation, Pine River Institute

When you are about to start measuring the impact of your work, you might first ask, ‘What measurement instrument is the best?’. Asking this question first, however, might result in your measuring factors that don’t resonate with your practice, or omitting factors that are important outcome indicators. The resources and culture shift required to implement evaluation are too costly for this. It’s vital to start right, which means a little bit of ground work. This article outlines some steps to start right, and ultimately, optimize your evaluation efforts.

Step One. Decide Why.

Why are you doing evaluation? Do you want to demonstrate your impact to clients, families, staff, accreditation, state, county or payor? Do you want to validate your treatment approach? Understand where to invest in professional development or make program changes? Engage a university research partner? Maybe you want to do all of this, and evaluation makes that possible, but write down your reasons for evaluation. This will be your guide and will keep you in scope.

Step Two. Get Curious!

‘Success’ is what you want to measure, and ‘success’ means something different for each of your stakeholders. Engage them! Ask clients (lots of them!), ‘if in a year you can say that treatment worked for you, what does that mean?’ Try not to prompt responses; you’ll be surprised with the indicators that are important to your clients. If they really don’t know, you might ask if success means they are working, schooling, medication-free, handling stress, relating to family, social, etc. But keep your input minimal.
Ask your staff what they work on with their clients. In what areas do they foster change? If they see a ‘success’ client in a year, what does that mean? Ask your executive team, ‘why are we here?’ ‘in what ways are we having an impact?’
If you plan to use your evaluation to broaden knowledge, you may want to explore ‘mechanisms of change’- why do you work, or what fosters treatment success. Ask clients ‘what was a moment of change for you?’; ask clinicians ‘what is your magic?’; ask executives ‘where do we expect treatment fidelity?’.

Step Three. List Key Indicators.

Using information from your environmental scan, list the common outcome indicators, and if applicable, process indicators. Outcome indicators are your success markers such as substance use, mental health, criminality, hospitalization, vocation, etc. Process indicators are the mechanisms of change such as treatment completion or progression, therapeutic alliance, family engagement, fidelity to a manualized approach, etc.

Step Four. Be a Journalist.

Who will complete the instrument (client, parent, clinician, referrer)? Where will it be completed (online, on-site)? How will it be administered (paper, device, in a group)? If there is a user cost, do you have the resources? Do you need quantitative responses or qualitative? When will it be administered (before, during, after)? Importantly, if you want to measure change, you measure indicators before and after treatment!

Step Five. Find Your Instruments!

You can explore many sources, such as university libraries and google scholar to see the thousands of measurement tools available. Your ground work will help you narrow the options, thankfully! OutcomeTools has several dozen instruments available to view.
Read every question! Have your stakeholders review different instruments. The tool must align with your language and your purpose. Look for problems such as double-barreled questions (asks two questions in one), hard to remember items (dosage of alcohol or substances), too much jargon, difficult time anchors, assumptive questions (e.g., does your child think about suicide), no ‘other’ or ‘don’t know’ option where it’s needed.

Step Six. Implement!

This is where the work gets real! Implementation is a huge undertaking, requiring a working culture shift, huge leadership support, and all-hands-on-deck. For some, evaluation will seem like just another task in their already overburdened day; demonstrate the benefits to them! Show clinicians the scores of one of their clients. Ask them if the scores resonate with their experience. Ask them about the client, the treatment approach, challenges, surprises, successes. Most often, clinicians are so happy to share their experience and see the results of their hard work.
One way to really engage all stakeholders in evaluation is to implement Feedback Informed Treatment, which is periodic measurements of key indicators that are used to inform and guide treatment. This is a powerful way to close the gap between research and practice. Feedback Informed Treatment is also a way to increase your post-treatment survey response rates. When clients see the results of evaluation during treatment, they are more likely to respond to requests for contribution after treatment.
If we can help you get started with these steps or have questions, please reach out to our expert staff at OutcomeTools – team@bestnotes.com or 866-543-6646.

For more information on this topic, please see our recent webinar here!