Introduction

Research and evaluation studies provide program administrators, policy makers, and service providers with information and insight into whether programs work and for whom. Rather than relying upon anecdotal evidence or intuition, programs are able to link services with performance measures and outcomes. The results can be used to revise or refine specific approaches, policies, and practices to ensure better outcomes for children and families.

There has been a long history of tension among researchers and service providers, however, with both often feeling disconnected from one another. Service providers and program administrators sometimes feel as if research is artificial or inapplicable, or that it applies to services that are materially different from what their programs offer. In addition, many programs lack the funding to cover the cost of evaluations, or sometimes evaluations are conducted as an afterthought to program design. It is important for researchers to conduct applied studies in real-world settings. At the same time, it is important also for service providers to be open to accepting results that may call for changes in their delivery methods or mechanisms.

Research has produced considerable, new information in recent years about both specific types of services, such as home visitation programs, school-based sexual abuse prevention programs, or parent education programs. In addition, some studies have focused on the effects of specific elements of programs, such as the intensity or length of services. However, it is difficult for program staff, who do not necessarily speak the language of researchers, to sift through the available research and determine the potential impact that research results may have on their programs. Bridging this gap continues to be a critically important challenge.

Though the notion of demonstrating impact through evaluation has made great strides, the maltreatment prevention community needs to continue to engender a results-based approach to management. Managing for results focuses an organization on its specific goals and objectives and entails the selection of appropriate performance measures and the use and the reporting of those measures for purposes of ensuring program accountability and promoting effective and efficient allocation of resources. This process should be engaged at the earliest possible point, as part of development and implementation of the entire program concept.

Though service providers increasingly recognize the importance of evaluation and performance measurement, and the connection between demonstrating impact and funding security, there is a present and understandable gap between the recognition and will of providers and their actual capacities. Thus, the field needs to continue to promote and encourage collaboration between service providers and local entities holding evaluation research capabilities, whether in arms of State and local government, universities, or other organizations. These connections are fundamentally important to professionals in the field who share a common interest in understanding the dynamics of what works in prevention and for whom.

Emerging Practices: A Pilot Test and Next Steps
The Emerging Practices in the Prevention of Child Abuse and Neglect project represents a first-ever deployment of a federally funded program nomination procedure specifically targeted toward programs for the prevention of child maltreatment. Working with a diverse pool of nominations, the Advisory Group met in October and November 2002 to review each nomination and to reach consensus regarding the final disposition and categorization of each nominated program. Nominations of programs submitted as "Effective" were reviewed with an emphasis on the quality of the methodological design and the integrity of the resulting program outcomes. For those submitted as "Innovative," reviewers looked for new, creative ideas and strategies for preventing child abuse and neglect.

The project faced considerable challenges, and there were important lessons learned for any future use of this methodology. One particular challenge for this project was the reality that prevention programs are diverse. The population of existing programs that should fall within the parameters of a project like this is not easily identifiable. The most difficult of all inclusion issues was whether the population of programs of interest to this process should include only those programs that define themselves as child maltreatment prevention programs. What about programs that are not specifically geared toward maltreatment prevention, but have the capacity to prevent maltreatment just the same by increasing family capacity, such as a parent-child reading program? The answer to this question drives a number of tasks that are critical to deploying any program nomination procedure, first and foremost of which is how national outreach and promotion of the project is conducted.

Under this first effort, the nomination process was intentionally designed not to be unduly restrictive, but rather to cast a wide net. The goal was to learn more about "what's out there" in terms of new programs on the prevention landscape, as well as to generate a sufficient number of nominations with which to work. Thus, the two principal nomination categories-Effective and Innovative-were broadly circumscribed in a way that would open the process to the maximum extent possible, leaving the burden of selection to the panel of expert reviewers.

During the course of the review, however, it became apparent that a substantial number of nominations did not meet the criteria for "Innovative" because of program age and activities, or the criteria for "Effective" due to inconclusive outcomes based on methodological design considerations. However, many of these programs either had interesting and unique aspects or had made a concerted effort to undertake research and evaluation with limited resources. The Advisory Group concluded that these programs had features that would be informative to the field. Consequently, the Advisory Group wanted to recognize the "noteworthy aspects" of those programs, especially those that made a good effort at evaluation and presented positive preliminary results, as well as programs with some unique aspect that could possibly be replicable or programs that could become candidates for more rigorous evaluation.

A careful review of the nomination categories utilized under this project will precede any future use of this methodology, with specific consideration given to clarifying and expanding the framework of categories to reflect the universe of prevention programs more precisely. This would require potential nominators to consider their programs against more specific requirements and standards, ensuring more unified, homogenous categories of programs for consideration. In addition, within categories, it may be possible to implement a numeric scale or rating system that would reduce interpretability and maximize objectivity, standardization, and interrater reliability.

A nomination process like this one is inevitably vulnerable to self-selection biases. While this report presents new information on numerous programs, it also is true that a number of widely known and respected programs were not nominated. The omission of major program models in this first-ever effort has implications for any future deployment of this methodology in terms of outreach and promotion. Any future use of the methodology may utilize an invitation procedure to ensure that known models are aware and have sufficient time to respond.

Finally, the review process was limited to information in accordance with submission requirements identified in the nomination instrument, supplemented, if available, by any reports containing supporting evaluative analysis. It is not altogether clear, however, whether all programs made the best possible case for their nominations. Some nominations, in fact, were not supported by sufficiently apparent internal logic in terms of the connections between what they do, why, and what they intend to accomplish. Site visits to programs were not possible in this initial effort, but will be considered in the future to collect additional information about programs in terms of services, delivery, setting, and participants.

The full report on the Emerging Practices project, Emerging Practices in the Prevention of Child Abuse and Neglect, can be found on the National Clearinghouse on Child Abuse and Neglect Information Web site:

HTML: nccanch.acf.hhs.gov/topics/prevention/emerging/report/index.cfm

PDF: nccanch.acf.hhs.gov/topics/prevention/emerging/report.pdf

A print copy of the report can be ordered by contacting the Clearinghouse at (800) FYI-3366, (703) 385-3206 (fax), nccanch@calib.com (e-mail).

© National Clearinghouse on Child Abuse and Neglect Information