How to Make AI Investment Decisions With Incomplete Data

Every AI vendor will hand you a beautiful slide deck full of projections. Your internal team will hand you a pilot with mixed results. Your CFO will ask for a number you don't have yet. And you still need to make a call by Friday. This is the actual job of executive decision-making in AI. Not waiting for perfect data. Not running another pilot. Making a defensible call with 60% of the information you wish you had.

Why the 'Just Run Another Pilot' Instinct Is Costing You

I spent three years at a Top 5 bank watching a payments AI initiative stall because leadership kept asking for 'more data.' Every quarter, someone requested another proof of concept. Another vendor comparison. Another risk review. By the time the team had what leadership considered 'enough' data, the competitive window had closed. A regional bank had already deployed a similar capability and was processing exception handling 40% faster.

The instinct to gather more information feels responsible. It feels like good governance. But in AI specifically, the data you want often does not exist until you commit resources. You cannot know your actual production accuracy rate from a sandbox pilot. You cannot predict adoption rates from a survey. You cannot model true cost-per-transaction until the system is running against real volume.

McKinsey's 2025 AI survey found that 74% of enterprise AI projects that stalled in pilot phase never recovered. They did not fail on technical merit. They failed because the decision window closed while leadership waited for certainty that was never coming.

The question is not 'Do we have enough data?' The question is 'Do we have enough signal to make a reversible decision?' Those are very different bars.

The Signal-to-Noise Framework for AI Go/No-Go Calls

After making dozens of these calls across three banks, I started using a framework I call Signal-to-Noise. It does not require complete data. It requires honest answers to five questions. Each one gets scored 1-5, and the total tells you whether to commit, pause, or kill.

Question one: Business pain clarity. Can you describe the problem this AI solves in one sentence that a business line leader would nod at? If the use case is vague or technology-driven ('we should use GenAI for something'), score it a 1. If a specific team is losing specific hours or dollars on a specific task, score it a 5. Most AI projects that fail score 2 or below here.

Question two: Data readiness. Not 'is the data perfect,' but 'can we access the data we need within 30 days without a multi-team governance battle?' At one bank, we killed a promising fraud detection project because the data lived across four systems with three different data owners and no shared taxonomy. The AI was ready. The data politics were not. Be honest about this one.

Question three: Pilot signal strength. If you ran a pilot, did it produce at least one metric that was clearly better than the current state? Not marginally. Clearly. A 3% improvement in processing speed is noise. A 35% reduction in manual review time is signal. If your pilot produced ambiguous results, that is actually useful information. It means the use case might not be strong enough.

Question four: Sponsor durability. Will the executive sponsor still be in their role in 12 months? Will they still care about this initiative if priorities shift next quarter? I have seen more AI projects die from sponsor departure than from technical failure. At one bank, we had four sponsor changes on a single AI initiative over 18 months. It never shipped.

Question five: Reversibility. If this decision is wrong, what does it cost to unwind? A $200K cloud-based deployment you can shut off is very different from a $3M on-premises installation with a 36-month vendor contract. Low reversibility demands higher signal on the other four questions. High reversibility means you can move faster with less certainty.

Scoring It: The Decision Thresholds That Actually Work

Add up your five scores. Maximum is 25. Here is how I read the results, and these thresholds have held up across roughly 30 AI investment decisions I have been part of.

Score of 20-25: Go. You have strong signal across the board. Commit budget, assign a delivery lead, set a 90-day milestone. Do not run another pilot. The risk of delay now exceeds the risk of action.

Score of 14-19: Conditional go. You have enough signal to proceed, but there is a weak spot. Identify the lowest-scoring question and build a 30-day gate around it. Example: if sponsor durability scored a 2, get the sponsor's boss to co-sign the initiative in writing before you spend a dollar. If data readiness scored a 2, fund a 30-day data access sprint before committing to the full build.

Score of 8-13: Pause, do not kill. Something material is missing. The most common pattern here is strong business pain (score of 4-5 on question one) but weak data readiness or pilot results. This is the zone where 'run another pilot' is actually the right call, but only if you define what specific signal the pilot must produce and set a hard deadline.

Score of 7 or below: Kill it. Reallocate the budget and the people. This is not a reflection of the technology. It is a reflection of organizational readiness. The best AI in the world will not survive a weak use case, inaccessible data, and a disengaged sponsor.

I used this framework to kill a $1.4M NLP project at a bank where I worked. The business pain was real (score of 4), but data readiness was a 1, the pilot produced no clear improvement over the rules-based system already in place (score of 2), and the sponsor was 6 months from retirement (score of 1). Total: 10. We killed it, reallocated $900K to a document processing initiative that scored 22, and that project shipped in 5 months with a 42% reduction in processing time.

The Three Traps That Wreck Good Decision-Making

Even with a framework, executives fall into predictable traps when making AI investment calls. I have fallen into all three.

Trap one: Sunk cost momentum. You have already spent $400K on a pilot and two vendor evaluations. Killing it feels like waste. It is not. The $400K is gone regardless. The question is whether the next $1M will produce a return. Score the initiative fresh, as if you were seeing it for the first time. If it scores below 14, the prior spend is irrelevant.

Trap two: Consensus addiction. You want IT, the business line, risk, and compliance all aligned before you move. That alignment will never happen organically on AI initiatives because each group has different risk tolerances and different definitions of success. You need a decision-maker, not a committee. At the banks where I saw AI actually ship, one VP owned the go/no-go call. Everyone else was consulted, but one person decided.

Trap three: Vendor confidence as a proxy for project confidence. Your vendor is very confident their solution will work. Of course they are. They are selling it. Vendor confidence is not signal. Internal pilot results are signal. Your team's ability to support the system post-deployment is signal. The vendor's reference customer who happens to be in a completely different industry is not signal.

The hardest part of executive decision-making with AI is accepting that you will sometimes be wrong. A 60% hit rate on AI investment calls is excellent. Most organizations are running at 20-30% because they either approve everything that looks promising or approve nothing because nothing looks certain. The framework above does not guarantee right answers. It guarantees you are asking the right questions before you spend.

Actionable Takeaway