September 2021Vol. 22, No. 8Strengthen Your State's Use of Data Evidence to Assess and Demonstrate Systemic Factor Functioning

Written by Steve Lao, M.P.H., child welfare data specialist, and Elizabeth Jones-Ferguson, M.S.W., child welfare data specialist lead, CWRP contractors for the Children's Bureau Child and Family Services Review team

The statewide assessment (SWA) is the first phase of the Child and Family Services Review (CFSR). During this phase, states assess the extent to which their child welfare system functions effectively to provide for the safety, permanency, and well-being of children and families whom the system serves. The assessment, done in collaboration with key partners and stakeholders, requires a comprehensive analysis of the state's programs, processes, and practices to determine the degree to which they produce desired results on the seven outcomes and seven systemic factors associated with the CFSR.

The assessment highlights the importance of having a shared vision that is reflected in the design of system structures and processes, as well as in the quality of practices and services delivered. Some questions to consider about the state's child welfare system in preparation for the SWA include the following:

Is the state's vision for its child welfare system well-communicated and well-understood?
Are system structures, programs, processes, and practices aligned with the vision?
To what extent do system structures and processes produce desired results?
How well does the state meaningfully engage key stakeholders-such as families and youth with lived experience in the child welfare system, judicial and legal communities, tribes, resource families, and services providers-in the design, operation, evaluation, and improvement process?
To what extent does the state's continuous quality improvement (CQI) system adhere to best practices in measurement and implementation?
How well does the state employ a systematic approach to understand and address potential disparities different populations experience that may contribute to inequity in services and outcomes for historically underserved populations?
How well does the state integrate and build on findings and improvement strategies across past and present federal and state plans and reports (e.g., Child and Family Services Plans [CFSPs], Annual Progress and Services Reports [APSRs], Court Improvement Program Strategic Plans)?

The SWA process underscores the necessity for having a robust and high-functioning CQI system to identify problems, develop and implement solutions, monitor and evaluate actions taken, and act upon the findings by scaling up, adjusting, or repeating the cycle. When CFSR round 4 kicks off, it will mark a decade since the Children's Bureau issued Information Memorandum 12-07 titled, "Establishing and Maintaining Continuous Quality Improvement (CQI) Systems in State Child Welfare Agencies" on August 12. It will also be more than a decade since the development of the national set of court performance measures for child abuse and neglect cases and the corresponding toolkit for courts to implement performance measurement. The federal guidance was a call for state child welfare agencies and courts to strengthen quality assurance activities and CQI systems to more effectively evaluate and strengthen operations and delivery of timely and quality services and improve child and family outcomes.

The SWA process is part of the ongoing federal monitoring and state CQI cycle and is intended to build on work states are already doing to identify system strengths and areas needing improvement, prioritize areas of focus, and make improvements through development, implementation, and monitoring of federal and state program improvement plans and the evaluation and reporting of progress. The following are two cornerstones of all these monitoring and reporting processes:

Reliance on quality and relevant data and use of evidence to evaluate and demonstrate outcome performance and systemic factor functioning
Broad and meaningful involvement from child welfare system stakeholders, partners, and persons with lived expertise

As states prepare for the round 4 SWA, it is helpful to assess the extent to which they conduct the following activities and, as needed, strengthen the quality of these processes. Embedding these activities into the routine of day-to-day business processes will help states build a strong foundation for the SWA:

Determine the availability of data-or the need to obtain or collect data-to understand system dynamics, outcome performance, and the routine statewide functioning of systemic factors
Routinely review state administrative data regarding the performance on the statewide data indicators and supplemental context data to assess system performance
Ask questions and follow up on focus areas using the wealth of additional quantitative and qualitative data housed by the agency, judicial and legal communities, service providers, and other system partners to help identify potential drivers of system strengths and challenges
Apply sound measurement principles in the development and use of data resources, such as knowing the questions that need to be answered, selecting the appropriate population for measurement, assessing data quality, and specifying measurement methods (e.g., measure, time period, population)
Determine which part of the CQI change and implementation phase the state is in for each focus area, and complete or revisit associated steps for that phase
Determine which combinations of data analyses provide the most compelling evidence to support the state's observation of system performance and systemic factor functioning and how best to articulate that in writing

In preparation for round 4, states will receive a revised SWA template and instructions, a framework for completing the assessment, data profiles and supplemental context data, and a series of systemic factor briefs that provide examples and suggestions to demonstrate the functioning for each systemic factor.

The Children's Bureau believes that after a decade of CQI system and performance measurement work, advancements in our knowledge of implementation science, and the introduction and application of the CQI change and implementation process, states are well-positioned to complete high-quality statewide assessments.

For more information or to request technical assistance with the process, contact your Children's Bureau Regional Office.

Additionally, the following are federal resources that are currently available: