The ubiquitous business acronym—ROI—is most commonly interpreted as Return on Investment; the natural expectation of a process improvement initiative, whether that applies to financial return or some other defined return metric, such as improved patient outcomes. Yet for many laboratories, even after major capital investments and expenditures, ROI ends up as rarely obtained information.

For major laboratory diagnostic system or instrument acquisitions, early-stage ROI is calculated based on cost projections and expected benefits. However, subsequent to acquisition, implementation, training, and go live, many such projects are considered concluded. The new system, instrument, or analyzer is incorporated into the workflow and once it becomes ingrained, there is often little or no systematic assessment as to whether the cost, quality, service level, or operational benefits expected—and put forth in an ROI— were achieved. Even in the event the hospital or laboratory management does seek to evaluate whether the assumptions of the original ROI have been met, it would be quite difficult to determine an accurate portrait of accomplishment unless some baseline standard had been set. Thus, the ROI process should be considered throughout the project phase, as firmly establishing that ROI has been met (or exceeded) will facilitate subsequent projects.

The Importance of ROI Oversight

Establishing a clear, detailed return on investment is important for a number of reasons:

• Money and budgets will almost always be tight
• The laboratory should expect to get what it pays for
• Laboratory leadership needs to know whether projected benefits occurred and if not, why

Given these realities, any truly successful capital acquisition will include metrics that measure and demonstrate that success. Due diligence should be exercised in ascertaining whether the projected ROI has been achieved, and if it has not, why. Fortunately, there are some straightforward steps that can be taken to assess and validate expected outcomes and ROI, as well as approaches to remediate any ROI shortfalls.

A Comprehensive Assessment

One approach is the use of a comprehensive operational benefits realization assessment (COBRA), a process for an independent, statistically valid analysis with evidence-based metrics that measure a capital project’s positive effects on key performance and quality indicators and assesses ROI achievement.

Throughout the new system or instrument acquisition, the COBRA process helps measure a number of important variables and key performance indicators (KPIs) that would constitute success in meeting objectives. After acquisition, implementation, operational use of the system, and system stabilization, COBRA then remeasures the same variables and uses statistical techniques to determine changes that can be attributed to the use of the new system or instrument.

Overview Steps in the COBRA Process

The following are essential steps in the COBRA process:

• Quantify the expected financial benefits versus costs and projected ROI
• Negotiate the vendor contract to commit to the expected ROI and benefits
• Assess parameters and KPIs associated with the system or analyzer to be replaced and project the expected improvements of those parameters and KPIs before starting implementation
• Identify other initiatives that might affect the new system and influence KPIs
• Complete implementation and allow time for stabilization
• Remeasure parameters, KPIs, and actual costs associated with the expected benefits of the system
• Statistically evaluate differences observed; are they significant and attributable to the new system or instrument?
• Were the results as expected or are there shortfalls?
• If shortfalls are determined, work with vendor to remediate

Establish Baselines and Audits

A strong ROI analysis will ultimately depend on the establishment of a pre-implementation baseline (PIB). This will serve as a means to determine the existing environment prior to implementation disruption. Its purpose is to identify the key parameters and KPIs that may be affected by the project. It also helps determine the projected resource and financial expenditures of the acquisition and implementation.

Subsequently, a post-implementation audit (PIA) is likewise essential to establishing a credible ROI. These audits serve as a means to determine the changed environment after the new system or instrument implementation and stabilized usage. Its purpose is the measurement of key parameters and KPIs that were affected by the project and validation of ROI.

Relationship of COBRA, PIB, and PIA

The COBRA and PIB methodologies provide useful knowledge about pre-project operations and represent effective methods for obtaining and measuring key parameters. Post-implementation and stabilization, the PIA obtains measurements of outcomes achieved relative to the earlier quantified KPIs. Together, these methodologies should enable a clear, statistically valid assessment of improvements in multiple measures of laboratory operations including, but not limited to:

• Quality
• Timeliness
• Productivity
• Patient safety
• Customer satisfaction
• Cost reductions

Leveraging Vendor Contract Negotiations

You can help ensure that the projected benefits indicated in the ROI will be met by a given vendor by including those expectations and objectives in any requests for proposal (RFPs) and subsequent contracts. Further, you can obligate a vendor to underwrite an independent COBRA project (performed internally or externally) with inclusion of a documented PIB when available as an addendum to the contract. Lastly, require the vendor to respond to any deficiencies identified in the PIA at no additional cost.

Pre-Implementation Baseline Actions

As an integral part of the PIB, list and define quantitatively and qualitatively the expectations and objectives for the new system or instrument. Identify and measure the characteristics of the system or instrument planned for replacement by this project. Such variables could include:

• Staff FTE(s) necessary for optimal operation of the new system or instrument
• Outreach clients supported by the system and associated test volume and revenue
• Internal and external costs to operate and maintain existing system or instrument (to be replaced by this project)
• Performance and quality measures (eg, TAT, missing or mislabeled specimens, various selected KPIs, etc)
• Further bridge instrumentation or middleware
• External system interface quality or service issues (frequency and duration)
• User satisfaction

Post-Implementation Audit Actions

The following basics of a PIA process can be adapted to a laboratory’s specific workflow and implemented technology:

• Delay the PIA until system stabilizes (eg, 6 to 9 months after go-live)
• Remeasure PIB variables
• Compare results with original expectations and objectives
• Identify shortfalls or unmet expectations or objectives
• Work with vendor for remedies, as necessary

The results of the initial COBRA findings and PIA can include validation that desired outcomes and ROI have been achieved and proof that the system or analyzer has made a substantive difference. Confirmation that the outcomes achieved were due to the system or analyzer itself (and not other events) may require more rigorous analysis (see FIGURE 1). To help determine if the changes obtained and measured are statistically meaningful and not due simply to chance, certain statistical methods may be brought to bear, including:

• Analysis of variance (ANOVA)
• Dependent t-test, Wilcoxon signed-rank test
• Analysis of covariance (ANCOVA)
• Multiple analysis of variance (MANOVA)