How to tell whether a software vendor can actually build it
Case studies and team bios do not distinguish vendors. Six questions that surface real engineering depth, and the answers that should end the conversation.
Every vendor's site looks the same. Case studies with impressive percentages, a team page, a list of technologies, a process diagram with four or five phases. None of it distinguishes a team that has run systems in production from one that has not, because none of it is falsifiable.
What does distinguish them is how they answer questions about failure. Success has many plausible narrators. Failure has only people who were there.
Six questions
Ask these of anyone you are seriously considering. The value is not in the specific answer, it is in whether the answer has texture.
1. Tell me about a project that went badly, and what you would do differently.
The answer to listen for is specific and self-implicating: a decision they made, why it seemed right, what it cost, what changed afterward. The answer that should worry you blames the client, blames a vendor, or describes a difficulty that was nobody's fault.
Everyone has had a bad project. A team that cannot name one is either inexperienced or not being straight with you, and neither is what you want.
2. What is your on-call rotation, and who is on it?
Teams that build systems and hand them off do not have one. Teams that operate what they build do, and they will tell you how it works without preparation, because they live it.
3. Walk me through your last production incident.
Detection time, who was paged, what the fix was, what changed afterward. A team that runs software has this story ready. A team that does not will describe a process rather than an event.
4. How do you handle a requirement that turns out to be wrong mid-project?
Every project has these. The answer reveals the contract model and the honesty. "We raise it, price the change, and let you decide" is workable. "That does not happen, our discovery is thorough" is a claim that will be tested and fail.
5. What would you refuse to build?
A vendor with no answer will build whatever is asked, including the thing that should not exist. The specific refusal matters less than having one.
6. Who exactly will work on this, and what else are they on?
The gap between the people in the pitch and the people on the project is the oldest problem in this industry. Ask for names, ask what percentage of their time you get, and put it in the agreement. If you want to put these to us directly, that is what talking to an engineer is for.
What to ask for instead of a portfolio
Portfolios are curated. These are harder to stage.
| Ask for | What it reveals |
|---|---|
| A code sample from real work, sanitized | Whether the code has tests, comments that explain why, and readable structure |
| A reference who left the relationship | How they behave when the engagement ends, which is when it matters most |
| A postmortem they wrote | Whether failures are analyzed or absorbed |
| Their deployment process, described end to end | Whether release is routine or an event |
| A paid discovery before the full commitment | Whether their thinking is worth what they charge, at low risk |
| Their answer on data access in production | Whether access is controlled or ambient |
That last row deserves emphasis. Ask who at the vendor can read your production data, how that access is granted, and whether it is logged. A vendor whose engineers all have standing production access is telling you their controls are informal, whatever the security page says. Ours are described in our security practices.
Signals that should end the conversation
Some answers are disqualifying on their own:
- A fixed price for a system they have not examined. Either the price is padded or the scope will be renegotiated. Both are your cost.
- No named engineers, only roles. You are buying capacity from a pool, which is fine if that is what you want and a problem if you were told otherwise.
- No opinion on your architecture. A vendor with no view has not thought about your problem yet.
- Compliance described as "we can handle that". For regulated data, either they have done it under a signed agreement or they have not. See what a BAA actually obligates.
- Reluctance to write down assumptions. Assumptions in writing are how a fixed scope survives contact with reality.
Start smaller than the whole thing
The most reliable evaluation is not an evaluation. It is a small paid engagement with a real deliverable, run before the large commitment.
Two weeks of discovery produces an artifact you can judge on its merits: does it show they understood the system, did they find things you did not know, is the plan something you would follow. And if the answer is no, you have spent a fraction of the budget finding out. What that engagement should produce is in what a two-week discovery must deliver.
The geography question is separate and often conflated with this one. It is covered in nearshore, offshore, or US.