NDARC Monograph No. 45 (2000)
EXECUTIVE SUMMARY
This paper provides an overview of the issues relevant to the development of a treatment outcome monitoring system in New South Wales.
The Australian National Minimum Data Set is a new, ongoing mandatory data collection system across Australia that commenced collection in July 2000. Within the state of New South Wales, a joint project of NDARC and the NSW Health Department established a more extensive state-wide treatment data set that also began collection in July 2000. The next phase of the NSW project is the addition of a brief, generic treatment outcome module. This monograph is a review of the literature on the processes and predictors of alcohol and other drug treatment outcome, and the monitoring of treatment outcome. This and a previous NDARC Technical Report reviewing the available treatment outcome measures will form the basis of the recommendations for the content of a brief outcomes module.
Several large studies have established the efficacy of treatment for substance use disorders. Over the past 30-40 years the alcohol and other drug field has accumulated considerable experience with the evaluation of treatment. This cumulative experience has been synthesized many times in order to highlight knowledge being gained over time and to suggest ways the knowledge can improve both the theory underlying various treatment interventions, and the cost-effectiveness of service delivery. A common theme across these major reviews has been the call for better designed evaluations of outcome; that is the assessment of changes in clients’ drinking and health and social functioning and the attribution of these changes to participation in treatment. Most of the knowledge we have gained about the treatment of substance use disorders has come from the evaluation of specific types of treatment activities (e.g., cognitive-behavioural therapy, pharmacotherapy, skills training), or treatment settings in which these interventions are delivered (e.g., inpatient versus outpatient, hospital versus community versus home). We have also learned, however, that a community needs a mix of interventions in order to meet the needs of a heterogeneous population, and that it is helpful to plan services along a “continuum of care.” This systems perspective to planning and delivering services has, in turn, increased the importance of evaluating the overall network of services that may be offered in a given jurisdiction.
Section 3 of the monograph draws the distinction between treatment research and outcome monitoring. Outcome monitoring is one of the predominant paradigms emerging within the broader frame of health services research. It has been described as a system that categorizes clients into homogeneous subgroups, establishes benchmark expectations of the services and outcomes for each group, then attempts to optimise a core set of outcomes by changing the process or services received.” Each element of this definition is fraught with conceptual and methodological challenges to be overcome and these are discussed in turn.
Section 4 outlines the critical features that should be included in the design of any given outcome monitoring system acknowledging that funding has a large impact on design. The monitoring system must sort through the array of possible measurement variables and decide on specific measures to include as outcomes, or as control variables in the analysis, as well as those that allow the comparison of homogenous subgroups of clients. The importance of baseline assessment at intake, and a reassessment (after some nominated period of time) is highlighted. Integration of the clinical assessment and baseline evaluation information is discussed, and it is planned that the brief instrument intended for outcome monitoring in NSW be integrated into current practice.
Section 5 outlines the methodological details to be decided on when planning and implementing an outcome monitoring system. These include:
- the addition of methods other than self-report to enhance the reliability and validity of outcome monitoring;
- method of selecting a sample of clients for follow-up from the whole treatment population, and the identification of the population of interest;
- size of follow-up sample needed in order to detect the size of differences between two groups on an outcome measure. This needs to take into account that only a proportion of the client population will consent to be followed up and that some of these may be ‘lost’ to follow-up;
- the timing and frequency of follow-up, including the start-date for the followup period, the duration of the follow-up interval, and the time period over which outcomes are assessed. A system for the successful follow-up of the maximum number of selected clients must also be decided; &
- other issues surrounding consent and implementation.
The choice of measures is critical to the success of an outcome monitoring system and is discussed in Section 6. It is widely accepted that a multi-measure approach is needed, although the choice of measures must be guided by the objectives that the treatment system is trying to achieve. Specific measures might be included for alcohol use, heavy alcohol use, and alcohol-related problems; drug use, heavy drug use and drug related problems; treatment utilization; crime and incarceration; employment/school problems; health and mental functioning; use of health and mental services; pregnancy and low-birth weight babies. In addition to the outcome measures, factors that help explain or predict outcome for certain groups of clients are also needed. Some measures may be used in one analysis as the outcome, and in another analysis as the predictor. Important client factors to measure as predictors are: severity of dependence; family and social supports; and psychiatric symptoms; as well as client demographic data and treatment process data.
Other issues guiding the selection of outcome measures should include the total time to be invested in the baseline assessment process, the potential use of the computer to assist in information collection, established reliability and validity data, cost of using the instrument (if applicable), follow-up data collection strategy and resources available for data collection, analysis and interpretation of information.
Sections 7 to 9 examine predictors of outcome specific to alcohol, opiates, cannabis use and psychostimulant use disorders respectively. Some treatments may be more applicable to certain substance categories (such as maintenance therapies for opiate dependence), and therefore may present different predictors of outcome. There is no reason to believe, however, that a brief, generic measure of treatment outcome would not be equally applicable across treatment and drug types.
The final section outlines the ways outcome data may be linked to utilisation and costs information. The primary objective of the outcome monitoring system is the gradual improvement of outcomes through changes to the delivery of services. Implied in this objective is improved cost-effectiveness of services, and it is crucial to develop an information system that links data on service utilization, costs and outcomes for specific sub-groups of clients. The development of costing protocols for substance abuse services that are appropriate for a large scale monitoring system is a very new development. These protocols must collect information from both an accounting and a societal perspective. Comparisons between providers must take into account that different providers may service clients with different characteristics.
The establishment of system boundaries is one of the first steps in the design and development of outcome monitoring systems. The boundary of the substance abuse treatment system can be difficult to determine, for instance many mental health services have specialized substance abuse counsellors. The substance abuse treatment services may be required to participate in a broader information system for mental health or broader health sectors.
The outcome monitoring systems described have developed largely in response to the failure of the traditional, clinical research paradigm in addressing the important questions that arise in delivering services along a continuum of care model. The future of outcome monitoring is filled with both facilitating factors and barriers. Outcomebased funding still meets with considerable resistance by some substance abuse providers. However, it is probable that there will continue to be a high level of demand for system-level evaluation and monitoring information.
In the Australian context, particularly in New South Wales, the ground is very fertile for the development of routine outcome monitoring. There have been significant increases in funding of alcohol and other drug services with a greater emphasis on accountability. Concurrently, there has been the introduction of the New South Wales Data Set that has standardised the collection of process data on clients. This has paved the way ideologically and logistically for the pragmatic collection of treatment outcome data as long as it is brief, valid and appropriate and firmly placed in the context of ongoing improvement of treatment processes.