Don't Sign the Six-Month Contract Yet: How to Run a Paid Pilot With a Software Development Agency

Why the Standard Vetting Process Keeps Failing

The standard agency vetting process looks rigorous: review the portfolio, check the case studies, call two references, meet the team on a video call, compare proposals. Businesses run this exact process and still end up six months and six figures into engagements that are visibly failing.

The reason is structural. Every artifact in that process is curated by the agency. The portfolio shows the best three projects out of fifty. The references are the two happiest clients they have ever had. The people on the sales call are the most polished communicators in the company — and frequently not the people who will write your code. You are evaluating a performance, not a working relationship.

What you actually need to know is unglamorous: Do they communicate when things go sideways? Is the code they write maintainable by someone else? Do their estimates mean anything? Do they push back on bad ideas or just say yes? None of that is observable from the outside. All of it is observable in two weeks of real work.

What a Good Pilot Actually Looks Like

A pilot is a short, paid, scoped engagement — typically two to four weeks — where the agency delivers a real, bounded piece of your actual project under normal working conditions. Not a coding test. Not a free spec. Not a discovery workshop. A genuine slice of production work, small enough to walk away from and real enough to be diagnostic.

Paid matters, and it protects you, not just them. Free pilots attract agencies with idle capacity and put the work at the bottom of their priority list — you end up evaluating how they behave when a project doesn't matter to them, which is the opposite of what you want to learn. Paying normal rates for a small scope means you observe their standard operating behavior: their real project management, their real communication cadence, their real engineers.

The right size is large enough to require coordination — at least two people on their side, a proper kickoff, some ambiguity to resolve — and small enough that the total cost is a rounding error against the full engagement. For most business software projects, that is a $5,000–$15,000 commitment to de-risk a six-figure one.

Choosing the Right Slice: Vertical, Real, and Slightly Ambiguous

The most common pilot mistake is scoping it as a throwaway — a demo, a prototype, a toy feature disconnected from the real system. Throwaway work produces throwaway signal. The agency's habits around testing, documentation, and architecture never activate because everyone knows the code will be discarded.

Instead, pick a vertical slice: one complete, self-contained feature that touches the real layers of your product — interface, business logic, data. A reporting export. An integration with one third-party service. One workflow automated end to end. Something you genuinely need, that ships to real users or real stakeholders, and that a future team (theirs or anyone else's) would build on rather than delete.

Deliberately leave one part of the scope underspecified. This feels counterintuitive — everyone tells you to write airtight requirements — but a small patch of ambiguity is the most diagnostic element of the entire pilot. Weak teams build the wrong thing silently or stall waiting for instructions. Strong teams surface the ambiguity in week one, ask sharp questions, propose options with tradeoffs, and document the decision. You are buying a preview of how they will handle the hundred ambiguities of the real project.

What to Measure While It Runs

Decide what you are grading before the pilot starts, and write it down — otherwise you will grade on vibes at the end, which is what the sales process already did. Five dimensions cover most of what matters.

Communication: Did you get a consistent cadence of updates without asking? When something slipped or surprised them, did you hear about it immediately or discover it at the demo? Silence during a two-week pilot predicts silence during month four of the real project.

Estimation honesty: Compare what they said in the kickoff to what happened. Missing an estimate is normal; concealing the miss is disqualifying. The teams worth hiring re-forecast openly the moment reality diverges from the plan.

Code quality under someone else's eyes: Have an independent developer — not the agency — review the deliverable. You are not looking for perfection; you are looking for tests, comprehensible structure, and documentation that would let a different team continue the work. If you do not have a technical reviewer available, a few hours of a fractional CTO's time for this review is money well spent.

Friction handling: Every real engagement has a moment of friction — a disagreement on approach, a delayed access credential, an unclear requirement. Watch what they do with it. Professionals resolve friction in the open; problem vendors either escalate it into drama or bury it.

Handover quality: At the end, they should hand you everything — repository access, deployment notes, decisions log — as if the engagement were ending forever. Because contractually, it might be. An agency that handles a pilot handover gracefully will handle the real relationship's boundaries gracefully too.

Structuring the Terms So You Can Walk Away Clean

The commercial structure of a pilot is simple, and its simplicity is the point. Fixed budget or tightly capped time-and-materials. A defined deliverable. Full IP assignment to you upon payment — the code, the documentation, the credentials, everything, regardless of whether you continue. And an explicit no-obligation clause: completing the pilot commits neither side to the larger engagement.

Any agency that resists these terms is answering your question early. Refusing IP assignment on paid work, insisting the pilot only happens bundled with a longer commitment, or pricing the pilot at a loss-leader discount that obligates you socially — each of these is the sales process reasserting control over an exercise designed to escape it.

One caveat in the other direction: respect that good agencies are also evaluating you. A serious firm will ask hard questions about decision-making authority, availability of your stakeholders, and access to your systems before agreeing to a pilot — because pilots fail just as often from client-side absence as from vendor-side weakness. An agency that interrogates your readiness is showing you seniority, not arrogance. The pilot runs in both directions, and the best outcomes happen when both sides treat it that way.

When a Pilot Is the Wrong Tool

Pilots are not universal. For very small projects — a few weeks of total work — the pilot is the project, and adding process around it is theater. For pure staff augmentation, where you are adding an engineer into your own team and workflow, a conventional trial period inside the engagement achieves the same thing with less ceremony.

And a pilot cannot substitute for basic due diligence. It tells you how a team works; it does not tell you whether the company will exist in two years, whether their senior people stay, or whether their contract terms are fair. Run the boring checks too. The pilot is the final and most reliable filter — it works best when the obviously wrong candidates have already been screened out, so you spend your two diagnostic weeks on a finalist worth testing.