What Your AI Vendor's “Case Study” Isn't Telling You

Last year, an AI vendor showed me a case study claiming their fraud detection model “reduced false positives by 80%” at a major financial institution. Impressive. I asked three follow-up questions. First: what was the baseline false positive rate? They did not know. Second: over what time period? “The initial pilot.” Third: did the number hold in production? Silence. The case study was based on a 6-week pilot with hand-selected data, and the vendor had lost contact with the client 4 months after deployment.

This is not unusual. This is the norm. AI vendor case studies are marketing documents disguised as evidence. They are designed to create confidence, not to inform decisions. And if you use them as your primary evaluation tool, you are making million-dollar bets on stories, not data.

The Seven Things Case Studies Never Include

After reviewing over 100 AI vendor case studies across banking, insurance, and healthcare, I have identified seven categories of information that are almost always missing. Each one is critical to your buying decision.

1. The real timeline. Case studies compress time. “Deployed in 12 weeks” usually means the vendor's work took 12 weeks. It does not include the 8 weeks of data preparation the client did before the vendor arrived, the 6 weeks of integration work after the vendor left, or the 4 months of tuning before the system reached the accuracy numbers quoted. I reviewed a case study that claimed a “90-day deployment.” The client told me the real timeline from contract signing to production-grade accuracy was 14 months.

2. The total cost. The case study will mention the contract value. It will not mention the internal engineering hours, the data preparation costs, the change management spend, the infrastructure upgrades, or the ongoing monitoring and retraining costs. A case study from a document processing AI vendor quoted a $200K implementation. The client's internal costs — data labeling, integration engineering, workflow redesign, and training — added another $340K. The real cost was $540K. The case study showed $200K.

3. The baseline. “Improved accuracy by 35%” means nothing without a baseline. 35% improvement from 50% accuracy means you are at 67.5% — still mediocre. 35% improvement from 90% means you are at 96.5% — genuinely impressive. Most case studies omit the starting point because the absolute numbers are less dramatic than the relative improvement. Always ask: improvement from what?

4. What happened after the pilot. The vast majority of AI case studies describe pilot results, not production results. Pilots run on clean data with dedicated support from the vendor's best engineers. Production runs on messy, real-world data with your team managing it. The performance gap between pilot and production is typically 15-30%. A model that hits 95% accuracy in a pilot often settles at 75-80% in production. The case study will never tell you this.

5. The team the client needed. Case studies describe the vendor's team — “our dedicated solution architects and ML engineers.” They rarely describe what the client needed internally. Did the client need two full-time data engineers for 4 months? A dedicated project manager? A compliance officer reviewing every model output? The internal team requirement is often the largest hidden cost, and it is invisible in the case study.

6. The edge cases and failures. Every AI system has failure modes. The fraud detection model that catches 95% of fraud also misses 5%. What is in that 5%? Are they low-dollar nuisance cases or high-dollar sophisticated attacks? The document classifier that processes 10,000 documents a day also misclassifies 200. Which 200? Case studies report the success rate. They never describe the failure profile. And the failure profile determines whether the system is safe for your use case.

7. Whether the client is still using it. This is the question vendors hate most. Of the 100+ case studies I have reviewed, fewer than 20% were confirmed to still be in active production use two years after the case study was published. Some clients switched vendors. Some brought the capability in-house. Some abandoned the initiative entirely. The case study is a snapshot of the honeymoon period. It tells you nothing about the marriage.

How to Actually Evaluate: The Proof-of-Value

Case studies are not useless — they tell you whether the vendor has worked in your industry and what problems they claim to solve. But they should never be the basis of a buying decision. Replace case study evaluation with a structured proof-of-value (PoV).

Step 1: Define success criteria before the PoV starts. Write down the specific accuracy, latency, throughput, and cost thresholds the system must meet. Get buy-in from all stakeholders. If the PoV does not meet these thresholds, you walk. No exceptions, no “close enough.” Vendors will push for subjective evaluation criteria like “stakeholder satisfaction.” Resist. You need numbers.

Step 2: Use your data, not theirs. The vendor will want to run the PoV on their demo data. Refuse. The entire point is to test the system against your data, with your edge cases, your quality issues, and your volume patterns. If the vendor cannot perform on your data, their case study results are irrelevant. Provide a representative sample — not your cleanest data, your real data.

Step 3: Measure total effort, not just results. Track every hour your team spends during the PoV — data preparation, integration work, meetings with the vendor, testing, and troubleshooting. Multiply this by the ratio of PoV data to production data. That gives you a rough estimate of the internal cost of a full deployment. If the PoV requires 200 hours of internal effort for 1% of your data volume, full deployment will require an order of magnitude more.

Step 4: Talk to the case study clients directly. Ask the vendor for references — and then ask the references the seven questions above. What was the real timeline? The total cost? What does the failure profile look like? Are they still using it? If the vendor will not provide references, that is your answer. If the references contradict the case study, that is also your answer.

A Real Example: Case Study vs. Reality

A Top 20 bank I advised was evaluating a customer service AI platform. The vendor's case study claimed: “Reduced call center costs by 40% with AI-powered resolution.” We ran a PoV and talked to the reference client. Here is what we found:

The case study said “40% cost reduction.” The reference client confirmed a 40% reduction in call volume routed to agents. But 60% of those AI-resolved calls resulted in customers calling back within 48 hours for the same issue. The actual sustained resolution rate was 16%. The case study measured deflection. The client needed resolution. Different metrics. Wildly different outcomes.

Our PoV on the bank's actual call data showed a 22% AI resolution rate — above the reference client's 16% but far below the case study's implied 40%. The bank negotiated the contract based on the PoV number, not the case study number. They saved $400K in Year 1 versus the pricing the vendor had initially proposed based on their “proven results.”

Actionable Takeaway