Last year, a Series B founder showed me an RFP he was about to send to seven agencies, including ours. I read it in three minutes and told him to throw it away.
Not because the project was bad ‚AI it was a clean, well-funded build with a sensible timeline. I told him to throw it away because his RFP didn’t ask a single question that would actually predict whether a vendor could deliver. He was asking the questions agencies are trained to answer. He wasn’t asking the questions we hope nobody asks.
I have been on the agency side for fourteen years. I have watched more than 500 projects ship across 20+ countries ‚ some that went beautifully, some that got rescued at month nine, and a handful I wish we had never signed. In that time I have sat in dozens of vendor selection meetings as the person being evaluated, and I have learned what separates a real procurement process from a security-theater one.
Here are the twenty questions I would ask if the table were turned and I were the buyer. They are grouped into the five things that actually predict outcome: who does the work, how you are priced, how the work is delivered, how it ends, and whether you should trust the people across the table at all.
If you take nothing else from this piece: skip past the certifications, the case-study deck, and the slick pitch. Use these twenty.
01:
Who actually does the work?
The single biggest variance in software project outcomes is not methodology, stack, or even budget. It is the specific human beings who write the code. Everything else is downstream of that. These four questions surface what an agency’s salesperson is least equipped to fake.
1. Who specifically will write my code ‚ name, seniority, GitHub?
In every losing pitch I have seen, the agency parades a “tech lead” who never touches the project again after kickoff. Ask for named résumés, with public GitHub or portfolio links you can verify. If the answer is “we’ll assign once you sign,” that means whoever is available the Monday after the PO clears. Push back. We always commit names in proposals because we have to. Any agency with disciplined delivery can.
2. What's the ratio of senior to junior on my project?
A team that is 80% junior with one senior architect overseeing them is not a bad team – at the right price. Most agencies bill that team at senior blended rates. Get the exact ratio in writing. Get years-of-experience averages. Match the ratio to the rate you are being charged and you will discover whether you are paying for a team or for a logo.
3. Where do your engineers physically sit, and what time zone overlap do they share with mine?
“We work in your time zone” usually means one project manager and a Slack channel that is responsive between 9 and 11 a.m. your time. Ask about engineering overlap, not coverage. Two hours of overlap between PST clients and South Asian engineering teams is normal and works; zero hours of overlap is a death sentence for any project that needs daily debugging.
4. What's your annual engineering attrition, and who owns my project if your lead leaves?
Engineering attrition above 30% is industry-normal in some markets and quietly catastrophic for any twelve-month engagement. Above 20% is a flag worth pressing. Ask explicitly: if my tech lead resigns in month four, who picks up, and how long is the ramp? An agency that has not thought about this question is an agency that has not lost a tech lead yet.
02:
How do you actually price this?
Pricing structure is where most software agency relationships poison themselves before kickoff. Fixed-bid contracts on ambiguous scope, change orders that compound monthly, and unit prices that look reasonable until you realize what they exclude – all of these are predictable, and all of them are visible in the first conversation if you know what to ask.
1. Fixed bid, time-and-materials, or capped T&M ‚ and which would you recommend for my scope?
A vendor that pushes fixed-bid on an ambiguous scope is optimizing for their margin, not your outcome. They have priced in 30% padding for the inevitable surprises, and you are paying for those surprises whether they happen or not. The honest answer for most discovery-light projects is capped time-and-materials with weekly burn reports. If your vendor cannot articulate why one model fits and another does not, they have not thought hard enough about your project.
2. What's your change-order process, and what triggered the last three change orders you billed?
Specific past examples beat any contract clause. Ask for three real ones, sanitized. If they cannot produce them, they either do not run change orders cleanly, or they refuse them entirely and absorb the cost – which sounds generous but actually means they price the risk into the original bid. Either way, you have learned something.
3. What do you bill for that I might not expect ‚ PMs, QA, DevOps, code reviews?
The gap between a $40-an-hour quote and a $90-an-hour quote is rarely about engineer quality. It is about what is bundled. Some agencies bill PM, QA, and DevOps separately; some include them. Neither approach is wrong, but if you are comparing quotes without normalizing for this, you are comparing nothing.
4. If we paused the project for 30 days, what's your hold cost?
This is a diagnostic question disguised as a logistics question. A real partner with a real bench has a real answer‚ usually something like a 25–40% retainer to hold the team. A vendor without that answer is either staffing your project from a freelancer pool or has no expectation of you ever pausing, which is a sales-stage thought, not a delivery-stage one.
03:
How do you actually price this?
Most agencies will give you the same Jira-board-and-daily-standup answer if you ask how they deliver. The questions below cut underneath the surface ritual and ask whether the actual engineering discipline exists.
1. Walk me through your last three project post-mortems — what went wrong?
If the answer is “nothing went wrong,” they do not do post-mortems, which means they do not learn, which means whatever went wrong on their last project is about to happen on yours. We have written hundreds of post-mortems at Ropstam. I keep a folder of the most painful ones because they have shaped our delivery process more than any framework ever did. Ask to see one, redacted.
2. What's your code review process, and can I sit in on one?
You are not auditing for security theater. You are checking whether code reviews actually happen, or whether the team commits straight to main on Friday afternoons. A vendor that will let you sit in on a review – on someone else’s project, with the other client’s permission‚ is a vendor with a real review culture. A vendor that resists this is hiding something.
3. What's your test coverage standard, and what's the coverage on your last shipped project?
“We follow industry best practices” is not an answer. A number is an answer. The honest range for most production projects is 60-80% unit coverage, with focused integration and end-to-end tests on critical paths. If they quote 95%, they are either lying or wasting your budget on tests that test nothing. If they cannot quote a number at all, they do not measure it.
4. Show me a real Jira or Linear board from a current project, with names redacted.
You will learn more from one screenshot of an actual workflow than from thirty slides of methodology. Look for the messy parts – the ticket that has been in review for nine days, the bug that was opened three months ago, the standup notes that show real disagreement. Sanitized boards are sales artifacts. Real ones are how the team actually operates.
04:
How does this end?
Every software project ends. Some end in production launch and a clean handover; some end in a quiet exit clause invoked by either side; a few end in litigation. The exit terms you negotiate at the start determine which of these futures you get. Most buyers never negotiate them at all.
1. Who owns the IP at every stage - pre-payment, mid-project, post-launch?
The default in many jurisdictions is not what you would assume. In some countries, the developer owns the code until full payment clears, even if you signed an assignment clause. Get IP transfer milestoned to each payment stage in writing. Ours is at every invoice; some agencies assign IP only at full payment, which is a leverage tool you do not want pointed at you.
2. What's the handover package on day one of go-live, code, docs, credentials, runbooks?
“We will document as we go” means you will receive a README and a cloud login. A real handover is a written runbook for every deployment, a credentials inventory with rotation dates, an architecture diagram that matches the current system, and a list of every third-party dependency with its renewal date and account owner. Specify this in the SOW or you will not get it.
3. What's your post-launch support model, and what's the SLA in writing?
“We’ll be there for you” has a 90-day half-life. Get the SLA in writing: response time for P0 versus P2, hours of coverage, what counts as a billable change versus a support fix, and what happens when the engineer who built the feature has moved to another project. A vendor that has not written this down for previous clients is a vendor improvising your support model.
4. If I want to move this project in-house in 12 months, what does that transition look like?
A vendor that flinches at this question is building lock-in, not partnership. A confident vendor will quote you a 30-to 60-day knowledge-transfer process, with shadowing and documentation handoff, and will price it cleanly. They know that some clients leave, and they would rather leave them well than badly. The badly-left clients become the worst reference checks of your career.
05:
Should I actually trust them?
The previous sixteen questions test whether a vendor is capable. The last four test whether they are honest. Capability without honesty produces a vendor who will deliver competent work and then nickel-and-dime you in change orders. Honesty without capability is a friendly disaster. You need both.
1. Can I talk to three clients - including one whose project went sideways?
Anyone can produce three happy references. The recovered-from-disaster reference is the one that tells you the truth. We hand out two of those every year, because every long-running agency has them, and the willingness to put the recovered client on the phone is the single strongest trust signal an agency can offer. If the answer is “all our clients are thrilled,” they have not been in business long enough to know better, or they are not telling you the truth.
2. What's a project you turned down in the last six months, and why?
Vendors who say yes to everything are not partners – they are staffing agencies with logos. We turn down roughly one in three inbound projects: wrong stack, unrealistic timeline, founder we do not click with, or a problem we have failed at before and have not yet figured out. An agency that cannot name a recent decline either does not have enough deal flow to be selective, or is selling you a service they should not.
3. What certifications, audits, or partnerships have you renewed in the last year?
SOC 2 Type II, ISO 27001, Microsoft Solutions Partner, AWS partner tiers, Clutch verification – these all expire. A current certification is a current company; an expired one is a press release. Ask for the issue date and the expiry. Ask for the auditor name. Most buyers do not, which is exactly why most agencies let them lapse.
4. What would you do differently if you were me, evaluating you against your two top competitors?
This is the single most diagnostic question in the entire list. A vendor that cannot answer it has not thought about why a buyer would pick someone else, which means they will not address those concerns in the proposal, which means they will lose the deals and never know why. A vendor that answers it honestly “we would be more expensive than X, and we are smaller than Y, here is when each of those matters” – is the vendor you can negotiate with in good faith. That is the vendor you want.
The one question that matters more than all twenty
If you ask only one question, ask this:
"If something goes wrong at 2 a.m. on a Saturday, who picks up the phone?"
Every contract clause, every SLA, every escalation matrix is an attempt to systematize the answer to this question. Most of them fail. What actually works is a small number of people who care enough to answer the phone. At Ropstam, that person is me. My cell number is on every active client’s contract. Most clients never use it. The ones who have, remember. Two of the three largest clients we have today started as a 2 a.m. phone call from a different agency’s former client whose vendor had stopped responding.
You are not buying a methodology when you hire a software partner. You are buying a relationship with a small number of humans who will be on the other end of a phone when something is broken and your weekend is on fire. The other nineteen questions on this list are tools for filtering out the agencies that cannot answer this one honestly. Ask it last. Watch what happens to the room.