What to watch out for in CRO work orders

“The single most important value creation activity in biotech is excellent clinical trial execution. Rapid trial enrollment, with the right inclusion/exclusions, with robust and consistently-measured endpoints, and delivered on time (all the way to database lock and top line data) – these are the things that unlock real value from the Science in startups.  After years in discovery, the clinic makes or breaks everything, so never cut corners or shortchange on an important early clinical trial.”

• Bruce Booth, Partner, Atlas Ventures, $2.7B assets under management

We’ve met with hundreds of translational scientists who have executed hundreds of trial work orders, and hear the same common problems:

1. Timelines that don’t account for procurement
2. Too short stability windows
3. Irreproducible assay results
4. Low bids to win, and then getting hit with “change orders”

Timelines that don’t account for procurement

“Panel validation in 4 weeks.” Seems reasonable, right? But make sure to check what this timeline includes. Does it account for the time getting reagents, the time to get special samples for validation, any kind of special counting beads, etc?

These aren’t small lead-time items. In our experience, if these are not accounted for, these could add more weeks of delay. Now, consider the direct costs. The mean direct cost to conduct a clinical trial is approximately $40,000 per day. A 4-week delay means $1.1M gone, before the first real data point was even generated.

Too short stability window

Your trial isn't a single experiment; it's run over 18, 24, even 36 months. Your CRO 'verified' the panel in January for Cohort 1. But it’s now October, and you're running Cohort 3. Did anyone prove that the frozen antibody cocktail they're using today performs identically to the one from 10 months ago?

Be sure to insist on stability data in a readable format. A lot of CROs will handwave this point. Make sure you are seeing data that reflects clinical reality.

Irreproducible assay results

The ultimate goal isn't just to make a panel verified,  It's to build a trial-grade assay with precision, so you can see true signal instead of noise from an assay. 

But our experience has shown that the typical assay variability (measured by coefficient of variation (CV)) from conventional CROs is often greater than 20%. This happens because their verification process is designed to simply “see a signal,” not to optimize for minimal variance.

An assay that has a 20% CV needs more patients (which meansmore time and $) to see the same signal to noise ratio as one that shows 3%.

Low bids to win, and then getting hit with “change orders”

A frequent tactic we hear about is a practice of bidding a “too good to be true” price to start a project, and to win your work. No sooner is the ink dry, than you get hit with change orders. And these orders can be multiple orders of magnitude larger than the original bid.

Ask for references during the buying process! These can save you a lot of heartache later.

Here are some questions you can ask of your CRO:

• Cost History: What is the historical average ratio of final project cost (including all change orders) to the original SOW bid price for projects similar to ours?
• Change orders: For your last projects of any kind, what percentage of them required one or more change orders, and what were the top three most frequent reasons for initiating a change order?

Wrapping up

As John Wooten said, “If you don't have time to do it right, when will you have time to do it over?”

The biggest risks to programs aren’t always science. It’s clinical trial execution. With developers heavily reliant on CROs for assay development, it’s important to pay attention to these details: actual validation time, stability of assay, and noise measurements.

CRO Work order risk audit

We’d like to offer you a complimentary, 30-minute CRO SOW Risk Audit. Send us your current or a past work order. We can benchmark against our market data and show you exactly where these hidden risks, in timelines, precision, and stability, are hiding in your own program.