June 2019Vol. 20, No. 5Testing and Piloting an Intervention for Implementation Success

Written by the Capacity Building Center for States

Imagine that a state child welfare agency is implementing a new kinship navigator program. After creating a theory of change, the agency selects a program model, adapts it to fit the needs of grandparents and other relative caregivers in the communities it serves, and creates tools for staff to use when providing information to caregivers and referring them to needed services. Imagine the state implements the program statewide without first testing the tools or piloting the program. Soon, staff discover that things are not working out as they had expected. Staff and relative caregivers are not completing referral forms correctly, and intake processes are overwhelmed. The agency is forced to backtrack.

Now, imagine that before statewide implementation, the state tests the forms with potential users and pilots the program in one or two divisions for a few months. The results are then used to "work out the kinks" before the program is adopted statewide.

Testing and piloting a program or other intervention can help agencies identify possible roadblocks, assess the target population's reactions to it, and make adjustments before implementing the program on a larger scale. These processes provide feedback loops to inform improvements and lay a foundation for successful and sustainable implementation.

The Capacity Building Center for States brief, Change and Implementation in Practice: Intervention Testing, Piloting, and Staging, can help child welfare agency leaders, managers, and implementation teams test, pilot, and stage an intervention to address an identified challenge. Some ideas from that brief are highlighted below.

Usability Testing

Usability testing is a process that helps teams quickly try out key program elements, assess their functionality, and refine or clarify, as needed. Usability testing provides a valuable opportunity to analyze and improve procedures and tools and sets a foundation for continuous quality improvement (Akin et al., 2013).

Usability testing follows a "plan, do, study, act" cycle, in which teams quickly prepare, implement, monitor, and gather feedback about a process or tool and then make corrections (see https://deming.org/explore/p-d-s-a for more information). The improved version of the process or tool may then be retested by repeating the cycle.

In a "plan, do, study, act" cycle, an implementation team performs the following (Permanency Innovations Initiative Training and Technical Assistance Project, 2016):

Creates a testing plan to help detect challenges and learn from experience (plan)
Puts the process, activity, or tool into place (do)
Gathers data and feedback about the process or tool and examines results (study)
Decides whether and where to make improvements (act)

Piloting

Pilot testing, or piloting, is a trial run of the implementation of a whole intervention (or significant parts of it) on a small scale (e.g., a single county or agency division) to identify problems and work out issues before launching it on a larger scale (Durlak, 2013). Piloting helps agencies decide whether they should continue to implement the intervention and, if so, how implementation should occur.

Piloting helps teams perform the following (Office of Adolescent Health, 2011):

Identify obstacles and adjust the intervention or the implementation plan before large-scale implementation
Gauge the target population's reaction to the intervention
Make decisions about allocating time and resources
Evaluate and measure results
Generate support and buy-in for the intervention by demonstrating early successes and making progress toward desired outcomes

Although they are similar concepts, usability testing and piloting differ in several ways. First, while usability testing focuses on one aspect of an intervention or implementation process (e.g., a tool or procedure), piloting involves implementation of the entire intervention or large portions of it, sometimes under carefully chosen conditions. Second, usability testing answers questions such as "Does the tool or process work?" and "Do the instructions make sense to staff?" On the other hand, piloting addresses questions such as "Was implementation successful?" and "Was the short-term outcome achieved?" Finally, while usability testing usually is a quick and targeted process, piloting is more comprehensive and may take longer to complete.

Taking a deliberate approach and using data to test and refine an intervention both help agencies establish a firm foundation for sustainable change. For more information on testing and piloting an intervention and other change and implementation topics, visit the Change and Implementation in Practice web page on the Center for States website.

References

Akin, B. A., Bryson, S. A., Testa, M. F., Blase, K. A., McDonald, T., & Melz, H. (2013). Usability testing, initial implementation, and formative evaluation of an evidence-based intervention: Lessons from a demonstration project to reduce long-term foster care. Evaluation & Program Planning, 41, 19-30. Retrieved from https://doi.org/10.1016/j.evalprogplan.2013.06.003

Durlak, J. (2013). The importance of quality implementation for research, practice, and policy. Retrieved from https://aspe.hhs.gov/report/importance-quality-implementation-research-practice-and-policy

Permanency Innovations Initiative Training and Technical Assistance Project. (2016). Guide to developing, implementing, and assessing an innovation: Volume 4: Initial implementation. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/guide_vol4_initial_implementation.pdf (1,978 KB)

U.S. Department of Health and Human Services, Office of Adolescent Health. (2011). Tips and recommendations for successfully pilot testing your program. Retrieved from https://www.hhs.gov/ash/oah/sites/default/files/pilot-testing-508.pdf (49 KB)