The ubiquitous business acronym—ROI—is most commonly interpreted as Return on Investment; the natural expectation of a process improvement initiative, whether that applies to financial return or some other defined return metric, such as improved patient outcomes. Yet for many laboratories, even after major capital investments and expenditures, ROI ends up as rarely obtained information.
For major laboratory diagnostic system or instrument acquisitions, early-stage ROI is calculated based on cost projections and expected benefits. However, subsequent to acquisition, implementation, training, and go live, many such projects are considered concluded. The new system, instrument, or analyzer is incorporated into the workflow and once it becomes ingrained, there is often little or no systematic assessment as to whether the cost, quality, service level, or operational benefits expected—and put forth in an ROI— were achieved. Even in the event the hospital or laboratory management does seek to evaluate whether the assumptions of the original ROI have been met, it would be quite difficult to determine an accurate portrait of accomplishment unless some baseline standard had been set. Thus, the ROI process should be considered throughout the project phase, as firmly establishing that ROI has been met (or exceeded) will facilitate subsequent projects.
The Importance of ROI Oversight
Establishing a clear, detailed return on investment is important for a number of reasons:
Given these realities, any truly successful capital acquisition will include metrics that measure and demonstrate that success. Due diligence should be exercised in ascertaining whether the projected ROI has been achieved, and if it has not, why. Fortunately, there are some straightforward steps that can be taken to assess and validate expected outcomes and ROI, as well as approaches to remediate any ROI shortfalls.
A Comprehensive Assessment
One approach is the use of a comprehensive operational benefits realization assessment (COBRA), a process for an independent, statistically valid analysis with evidence-based metrics that measure a capital project’s positive effects on key performance and quality indicators and assesses ROI achievement.
Throughout the new system or instrument acquisition, the COBRA process helps measure a number of important variables and key performance indicators (KPIs) that would constitute success in meeting objectives. After acquisition, implementation, operational use of the system, and system stabilization, COBRA then remeasures the same variables and uses statistical techniques to determine changes that can be attributed to the use of the new system or instrument.
Overview Steps in the COBRA Process
The following are essential steps in the COBRA process:
Establish Baselines and Audits
A strong ROI analysis will ultimately depend on the establishment of a pre-implementation baseline (PIB). This will serve as a means to determine the existing environment prior to implementation disruption. Its purpose is to identify the key parameters and KPIs that may be affected by the project. It also helps determine the projected resource and financial expenditures of the acquisition and implementation.
Subsequently, a post-implementation audit (PIA) is likewise essential to establishing a credible ROI. These audits serve as a means to determine the changed environment after the new system or instrument implementation and stabilized usage. Its purpose is the measurement of key parameters and KPIs that were affected by the project and validation of ROI.
Relationship of COBRA, PIB, and PIA
The COBRA and PIB methodologies provide useful knowledge about pre-project operations and represent effective methods for obtaining and measuring key parameters. Post-implementation and stabilization, the PIA obtains measurements of outcomes achieved relative to the earlier quantified KPIs. Together, these methodologies should enable a clear, statistically valid assessment of improvements in multiple measures of laboratory operations including, but not limited to:
Leveraging Vendor Contract Negotiations
You can help ensure that the projected benefits indicated in the ROI will be met by a given vendor by including those expectations and objectives in any requests for proposal (RFPs) and subsequent contracts. Further, you can obligate a vendor to underwrite an independent COBRA project (performed internally or externally) with inclusion of a documented PIB when available as an addendum to the contract. Lastly, require the vendor to respond to any deficiencies identified in the PIA at no additional cost.
Pre-Implementation Baseline Actions
As an integral part of the PIB, list and define quantitatively and qualitatively the expectations and objectives for the new system or instrument. Identify and measure the characteristics of the system or instrument planned for replacement by this project. Such variables could include:
Post-Implementation Audit Actions
The following basics of a PIA process can be adapted to a laboratory’s specific workflow and implemented technology:
The results of the initial COBRA findings and PIA can include validation that desired outcomes and ROI have been achieved and proof that the system or analyzer has made a substantive difference. Confirmation that the outcomes achieved were due to the system or analyzer itself (and not other events) may require more rigorous analysis (see FIGURE 1). To help determine if the changes obtained and measured are statistically meaningful and not due simply to chance, certain statistical methods may be brought to bear, including:
Remedial Actions and Caveats
If the measured outcomes are less than projected, it will be necessary to document the identified deficiencies and provide the document to your vendor. After the vendor has had the opportunity to review this assessment, arrange a meeting for discussions. It is possible that many of the deficiencies may be corrected by a vendor analyst and not require an additional expense. Otherwise, it may be necessary to invoke the terms and conditions of the executed contract to obtain resolution.
It may not be reasonable to expect the vendor to remediate or compensate for all identified deficiencies. Some may be within your own processes and procedures and not a fault of the new system or analyzer. The PIA results may not be notably precise, but done well, they should represent reasonable measures of ROI and a realization of the objectives of the investment.
Impediments to a PIA Analysis
As noted, rarely are retrospective ROI analyses properly performed, and there are a number of reasons for this major omission:
A COBRA project can help ensure that the promises of a technology acquisition project or other major capital expense can be and have been accomplished. Moreover, establishing a track record for diligent and comprehensive project performance bodes well for future endeavors. Finding efficiencies in the system and technology acquisition process is an ongoing effort, but one that yields other benefits including administrative confidence. Gaining a full picture using evidence-based metrics will sharpen the ROI process and never leave you with unobtained information.
Dennis Winsten, MS, FHIMSS, FCLMA, is president of Dennis Winsten & Associates, a health care consulting firm specializing in laboratory information. With over 25 years’ experience in the application of computer systems to health care, his professional affiliations include Fellow, Healthcare Information and Management Systems Society (HIMSS); Fellow, Clinical Laboratory Management Association (CLMA), CLMA Board of Directors, 2011–2013 and 1990-1993; Association for Pathology Informatics (API); and Clinical and Laboratory Standards Institute’s (CLSI) area committee on automation and informatics.