A threshold crossed
R&D, innovation and insights functions are organized around a long-standing constraint: evidence has been slow and expensive to produce. Stage gates, annual planning cycles, commissioned studies and a reliance on a few senior experts are rational responses to that constraint. Together they make up the function's protocol: how the work is divided, who does each part, what evidence a decision needs and when decisions get made.
The constraint is loosening fast. Models can now read the literature, lab records, patents and consumer conversations a function runs on, propose what to test next, interpret the result and explain their reasoning, and each new generation does more of this, more reliably, at lower cost. The protocol built for scarce evidence is becoming the main limit on what these functions can achieve.
The pressure to put this capability to work is considerable. Economists at Stanford and MIT estimate that US research productivity has declined by about 5% a year for decades, so each advance takes more researchers and more money than the one before. In McKinsey's research, more than 80% of executives rank innovation among their top three priorities, and fewer than 10% are satisfied with their organization's innovation performance.
Closing that gap takes more than capability. Added to the existing protocol, AI tends to make each step faster while the decisions, and the results, stay much the same. We expect the functions that pull ahead to be those that redesign the protocol around what the technology can now do.
From projects to outcomes
The existing protocol typically organizes work by project and stage. The next one organizes it as a hierarchy of outcomes, decisions and tasks, each defined precisely enough to measure.
Outcomes come first. They are the quantities the function is accountable for improving: risk-adjusted portfolio value, hit rate at launch, cost and time per successful formula, the share of research that changes a decision. Each needs a definition and a baseline before anything is built, because every later choice in the system is judged against it.
Decisions sit beneath outcomes. Each outcome moves through a set of recurring decisions: which concepts advance to quantitative testing, which formulation goes to stability, which research request is funded, whether a launch needs intervention. Decisions are where much of a function's expertise turns into value, so they become the unit the system is designed around. A system built for a named decision can be measured against that decision. A system built around a team or a workflow is much harder to measure.
Tasks and subtasks sit beneath decisions. A concept advancement decision, for example, rests on tasks such as sizing the opportunity, checking the concept against past projects and patents, estimating cost to make, and reading early consumer response, and each of those breaks into subtasks. This is the level where the work gets redesigned. Some subtasks exist largely because evidence used to be scarce, such as assembling a gate pack by hand or re-reading old studies to find a prior result, and they are candidates for retirement. The rest are assigned to agents, to people, or to both.
Choose an outcome
A decision that moves it
Which concepts advance to quantitative testingOne of several recurring decisions that move this outcome
Tasks behind the decision
- Size the opportunityAgent acts
- Check the concept against past projects and patentsAgent acts
- Estimate cost to makeAgent proposes
- Read early consumer responseAgent proposes
- Advance or stop the conceptPerson decides
- Assemble the gate pack by handRetired
A decision that moves it
Which formulation goes to stabilityOne of several recurring decisions that move this outcome
Tasks behind the decision
- Search past experiments and failuresAgent acts
- Check approved materials and supply riskAgent acts
- Propose the next experimentsAgent proposes
- Predict stability from history and early resultsAgent proposes
- Release the formulation to stabilityPerson decides
- Compile results from lab notebooks by handRetired
A decision that moves it
Which research request is fundedOne of several recurring decisions that move this outcome
Tasks behind the decision
- Tie the request to the decision it informsAgent acts
- Find existing studies that already answer itAgent acts
- List the untested assumptionsAgent proposes
- Propose the smallest study that resolves themAgent proposes
- Fund or decline the studyPerson decides
- Re-read old studies to find a prior resultRetired
A decision that moves it
Whether a launch needs interventionOne of several recurring decisions that move this outcome
Tasks behind the decision
- Read sales, reviews and research togetherAgent acts
- Compare results with the original forecastAgent acts
- Diagnose what is driving the gapAgent proposes
- Recommend an interventionAgent proposes
- Decide whether to intervenePerson decides
- Build the weekly tracking deck by handRetired
Principals and delegates
In the next protocol, people act as principals and agents act as delegates. Principals set objectives, own the decisions where expert judgment is contested, resolve conflicts between agents and between functions, and supervise the system. Delegates carry out subtasks within the permissions they are given.
Each subtask runs in a mode that sets who acts and how the work is checked.
Threshold to change modeThe eval that crosses it
Person decides
- Who acts
- A person, using evidence the agents assembled
- How it is checked
- Peers and the decision record
- When it applies
- Errors are expensive or irreversible, or experts themselves disagree
Typically stays here: funding a concept, releasing a formula to the plant, committing a team to a research direction.
Agent proposes
- Who acts
- Agent prepares, a person approves
- How it is checked
- The approver, with the evidence attached
- When it applies
- Errors cost more, or the agent’s record is still being established
Often starts here: proposing the next experiment, drafting a business case.
Agent acts
- Who acts
- Agent
- How it is checked
- Automated evals, plus human review of a sample
- When it applies
- Errors are cheap and reversible, and the agent has a proven record on this class of case
Often starts here: tagging interview transcripts against a research framework, screening literature, ranking a lab queue.
The assignment changes over time. When the evals show an agent matching expert performance on a class of cases, that class can move up a mode. When performance drifts, because the market shifts or the model changes, it moves back down. Autonomy is earned and revoked on evidence, which gives leaders a defensible answer when asked how far they trust the system.
The next intake begins in February 2027. Apply for the 2027 intake
The harness
The work model says what should happen. The harness is the system that makes it happen.
What should happen?
The work model sets the outcomes the function is accountable for, the recurring decisions that move them, and the tasks behind each decision, each assigned a mode. Everything below it exists to carry that work out.
What happens next, and who does it?
The orchestration layer runs each decision as a durable workflow. It holds state across steps that may take weeks, retries what fails, pauses where the mode requires a person’s approval and resumes when it arrives. A policy engine applies the mode and permissions for each subtask, and routing sends routine extraction to cheaper models while reserving frontier models for reasoning. The layer can also take over much of what the calendar does today: decisions are triggered when evidence crosses a threshold the principal has set, such as a project’s risk-adjusted value falling below its remaining cost. Every run is traced step by step, so any output can be followed back to the model, prompt, data and evidence that produced it.
Who does the work?
Each agent is scoped to a subtask, with defined tools, data access and permissions, and returns structured output against a schema. Models are chosen per task: a reasoning model to plan a study, a smaller model to extract data from thousands of documents, a vision model to read a chromatogram or a shelf photo. Some work calls for other instruments entirely. Cost roll-ups and stoichiometry run as deterministic code. Next-experiment selection often runs on a surrogate model, such as a Gaussian process inside a Bayesian optimization loop, which weighs a candidate’s expected performance against what testing it would teach, and can balance several objectives at once: performance, cost, footprint and supply risk. Any of these models can be swapped as better ones appear.
How do we know the work is good?
Evals hold every agent to a standard written by the function’s own experts, and they decide which mode each subtask runs in. Rubric, preference and outcome tests combine, and model graders are trusted only where they agree with the expert panel. They are the subject of the next section.
What gets handed over?
Briefs, concepts, formula proposals, study plans, findings and business cases are typed objects with defined schemas and version histories. Each handoff, between steps or between functions, becomes a contract validated on arrival: a brief entering R&D without testable targets for performance, cost and claims is returned before anyone starts work. Because each artifact records the evidence and the runs that produced it, a launch can be traced back through every artifact to the signal that started it.
What does the system know, and where did it come from?
The data layer connects the systems the function already runs on: electronic lab notebooks and lab information systems, product lifecycle and specification systems, research repositories, syndicated and panel data, social and search signals, sales. Above them sits a semantic layer, a model of the function’s entities (project, concept, formula, material, study, finding, decision) and the relationships between them. Entity resolution does much of the work here, since the same raw material can carry different codes in the lab notebook, the specification system and procurement. The layer is also versioned in time, so the system can reconstruct what was known on any past date, which is what makes an honest back-test possible. The systems of record stay where they are.
What has the function learned?
Memory holds the decision record: each decision with its assumptions, the forecast made at the time as a probability or a range, and the outcome when it arrives. It holds evidence with its source, method, sample and a shelf life suited to its type. And it holds the reasoning behind expert calls, so the logic of a decision can outlast the people who made it.
Outcomes flow into memory, retune the evals, and change how much each agent may do on its own.
Judgment as a test suite
In these functions, the quality of an output is often a matter of expert judgment: whether a concept is distinctive, whether a hypothesis is worth a bench week, whether a synthesis of forty interviews says something true about the market. In our work, building the agents has usually been the more tractable part. Encoding that judgment well enough to hold agents to it is where much of the design effort goes.
The eval layer combines rubric, preference and outcome tests.
Rubric evals
Break the standards senior experts apply into specific, checkable criteria, each anchored by examples of passing and failing work, so every grade can be explained.
Preference evals
Ask experts to compare two outputs and choose the stronger one. Where quality is debated, relative judgments tend to be more consistent than absolute scores, and a ranking model such as Bradley-Terry aggregates them into a scale of what the function considers good.
Outcome evals
Test the system against decisions whose results are already known, through back-testing against the company's own record.
Model graders make this affordable at scale, since expert time is limited. A model grader is used only for criteria where its agreement with the expert panel, measured with a statistic such as Cohen's kappa on a held-out set of graded examples, approaches the agreement experts reach with each other. That agreement is measured again whenever the grader's underlying model changes.
The same statistic, applied among the experts, separates settled judgment from contested judgment. Where senior people grade consistently, the standard is clear enough to hold an agent to. Where they diverge, the disagreement is recorded and the decision stays with a person. Mapping that boundary can be the first time a function sees its own judgment laid out case by case.
Back-testing on your own record
Outcome evals rely on back-testing. A company's history of concept tests and launches, formulations that passed or failed stability, and forecasts that met or missed can serve as a test set with known answers. The system runs on past cases with the outcome hidden, against a point-in-time snapshot of the data layer that contains only what was known on the date of the decision. That condition matters: a back-test that can see later publications, patents or sales is likely to overstate how the system will perform live. Run with a strict as-of cut, a back-test gives experts evidence from their own record, which tends to carry more weight with a skeptical formulator or insights director than a vendor demo.
The suite keeps growing. Every failure in production becomes a new test case, and the full suite runs in the deployment pipeline before any change to a model, an agent or a prompt ships, the way a software team runs regression tests. In production, the system samples live outputs for grading and watches for drift: shifts in input data, model behavior or the market that pull performance away from what the back-test showed. That discipline lets a function adopt a better model soon after it appears while protecting the standards it has built.
Memory that compounds
In many functions, much of what the organization learns isn't kept in a form anyone can use. A business case is approved, the product launches, and the forecast may never be compared with what happened. A study informs a decision, and two years later it can be hard to say which study it was or what it assumed. When a senior scientist retires, much of the reasoning behind years of formulation choices can leave too.
The memory layer records every decision as a structured entry: the options considered, the evidence used and its version, the forecast made at the time with its confidence, the owner, the mode it ran in, and the outcome when it arrives. Evidence carries an expiry suited to its type. A read on category trends may age in months, a material property may hold for years, and a price sensitivity estimate may not survive a competitor's launch. When evidence expires, the decisions that depend on it are flagged for review.
This record closes the loop that runs through the whole protocol. Outcomes are scored against the forecasts behind them using scoring rules that reward forecasts that are both accurate and honest about their uncertainty. Calibration curves then show which signals predict success, which models are well calibrated, and where the function's forecasts, human and machine, run optimistic or pessimistic. That evidence feeds the evals, the evals reset the mode each subtask runs in, and the orchestration layer routes the next decision accordingly.
- Step 1 of 6
A decision is made
The decision is logged with the options considered, the evidence used and its version, the forecast with its confidence, the owner and the mode it ran in.
- Step 2 of 6
The outcome arrives
The result lands in memory against the decision that produced it, and evidence past its shelf life is flagged.
- Step 3 of 6
Forecasts are scored
Each forecast is scored against the outcome, in a way that rewards forecasts that are both accurate and honest about their uncertainty.
- Step 4 of 6
Calibration is read
Calibration curves show which signals predict success, which models are well calibrated, and where forecasts, human and machine, run optimistic or pessimistic.
Illustrative. A curve below the diagonal means forecasts ran optimistic. - Step 5 of 6
Evals and modes reset
That evidence feeds the evals, and the evals move each class of cases up or down a mode.
- Step 6 of 6
Next decision routed
The orchestration layer sends each subtask to an agent or a person under the updated modes.
Each cycle leaves the function with better evidence, better-calibrated agents and a clearer record of its own judgment. That accumulated record is specific to each company, and it is the part of the system hardest for a competitor to replicate.
Where builds break
In our experience, AI for these functions fails in recognizable ways, usually when part of the protocol was skipped.
-
The system automates the artifact, producing a faster deck, report or gate pack, while the decision it was meant to serve stays the same.
-
Agents then meet a standard the function’s experts don’t recognize as their own, and those experts may start working around them.
-
Without a measure of the decision before go-live, the gain is hard to demonstrate, and the work becomes vulnerable at the next budget review.
-
A system tested with access to information from after the decision can look strong in review and underperform in production.
-
Agents are given decisions before their record supports it, an early error damages trust, and the program can retreat further than it needs to.
-
A separate tool that experts must remember to open tends to lose out to existing habits. The harness works best inside the systems, data and assistants the team already uses.
Choosing the first decision
The choice of first decision deserves as much care as the build itself. Good candidates recur often, resolve quickly enough to score, have a history of past cases to back-test on, and sit upstream of other decisions. Concept screening, research prioritization and next-experiment selection often meet all four tests. Portfolio strategy, which is made rarely and resolves slowly, is usually a better place to arrive than to start.
Pick the decision you are weighing
Answer the four tests
- Does it recur often?
- Does it resolve quickly enough to score?
- Is there a history of past cases to back-test on?
- Does it sit upstream of other decisions?
Where to start
Pick a decision and answer the four tests to see where to start.
Building it
New Protocol builds AI systems for R&D, innovation and insights functions on this protocol. Our strategists come from these functions and our engineers build production AI systems, and both work as one team. We start with one outcome and the decision that moves it, back-test against the client's own record, and expand from there.
Every system runs in the client's cloud, under its security rules, on the frontier models it chooses, and is used from the assistants its teams already work in. The client owns what we build: the code, the evals and the agent instructions.
New clients join in intakes, with a limited number of places in each. The November 2026 intake is closed. The next begins in February 2027.
Sources
- Research productivity declining about 5% a year: NBER Digest on Bloom, Jones, Van Reenen and Webb; American Economic Review, 2020
- Innovation priority and satisfaction: McKinsey, Innovation
- $360B to $560B in estimated annual economic value from using AI to accelerate R&D: McKinsey, The next innovation revolution, powered by AI, June 2025
Sections 4 to 8
Read the full protocol
Learn how to build the system, test it against your own track record and make it improve with every decision. Then see where builds break and how to choose your first decision. Add your name, function and work email to keep reading.