Most vendor evaluation checklists in circulation were written around 2019. They score references, hourly rates, and tech-stack overlap — and give zero weight to the variable that now decides delivery speed: how well a partner actually uses AI. If your scorecard can't tell an agentic delivery pipeline from a ChatGPT license, you're evaluating with the wrong instrument.
Why the old checklists fail
The 2019 template assumed throughput scaled with headcount. It no longer does. A senior team with real AI tooling — internal evaluation harnesses, agentic pipelines, institutional memory — outships a body-shop three times its size. We see this on every engagement running on Aura OS, our internal AI operating system. The old checklist can't see any of this. It asks how many developers you get, not how much verified output.
The old checklists also under-weight compliance. GDPR is table stakes; MiCA, DORA, and the EU AI Act now shape what a partner can legally build and where the data can live. A vendor who can't answer data-sovereignty questions in the first call will cost you the answer later, in production.
What's inside the full scorecard
The full playbook is a working document, not a think piece:
- A weighted 7-dimension scorecard — AI maturity, engineering depth, security posture, compliance (GDPR/MiCA), infrastructure ownership, delivery model, and total cost of ownership — with weights you adjust to your risk profile.
- Scoring rubrics for every dimension — what a 1, 3, and 5 look like in practice, so two evaluators land on the same number.
- A fill-in comparison sheet — score up to five candidate partners side by side.
- Red-flag triggers — answers that should end the conversation, including the ones that sound impressive.
Preview: the AI-maturity questions
A sample of what the full rubric asks every candidate partner:
- Do you run evaluation harnesses on AI-generated code, or does it merge on vibes?
- Where does AI-assisted work on our IP execute — whose infrastructure, whose retention policy?
- Show us a delivery pipeline artifact from a real engagement. Not a slide.
- What did AI not speed up on your last project? (No honest answer here means no honest answers anywhere.)
The full rubric covers ten questions per dimension, seventy in total. Fair warning: we score ourselves against it too, and it's not a scorecard we pass by default on every axis — no partner does. That's the point of weighting it to your priorities. For context on how we handle the infrastructure-ownership dimension, see our delivery model and how we run managed dev infrastructure to keep client cloud burn low.
Get the full scorecard. The complete weighted rubric, fill-in sheet, and all 70 evaluation questions are available on request — no charge, no drip campaign.
Tell us what you're evaluating and we'll send the full scorecard — request it with the form below; we reply within one business day.