AI Support Automation with RAG
Case study: building a RAG-powered support automation system for a Series B B2B SaaS platform.
Reviewed by Igor Gazivoda, Founder & CEO of StepTo · Published · Updated
Key Results
StepTo built a retrieval-augmented generation (RAG) system for a Series B B2B SaaS platform (~80 employees) whose support team could no longer keep pace with ticket volume. The system ingests product docs, past resolved tickets, and internal runbooks into a pgvector store, with an LLM-backed triage layer that drafts first-response answers and routes edge cases to human agents; routine tier-1 questions it can answer with high confidence are resolved automatically, and every other AI draft is reviewed by a support agent before it sends. The build took 4 months with a 3-person dedicated squad (1 ML/AI engineer, 1 backend engineer, 1 tech lead). Results measured over the following quarter: average first-response time dropped 70% (from ~5.5 hours to under 100 minutes), 38% of tier-1 tickets were fully auto-resolved with no human touch, and average handle time for tickets still requiring a human agent fell 45%.
From 5.5-Hour Ticket Backlogs to 70% Faster First Response
A Series B B2B SaaS platform had scaled its customer base far faster than its support team. Tickets piled up, first responses routinely took hours, and support engineers kept re-answering the same questions because the real answers lived scattered across Slack threads and Notion pages, not in anything searchable at the point of need.
What Results Did the RAG System Deliver?
StepTo measured these results over the quarter following full production rollout
What Problem Was the Client Facing?
StepTo's client, an ~80-person Series B B2B SaaS company, had grown ticket volume faster than headcount for three straight quarters. Every ticket was triaged manually, and average first-response time had crept up to roughly 5.5 hours, well outside what their enterprise SLAs allowed.
The deeper problem was institutional knowledge. Product docs existed but were incomplete, and the real answers to most recurring questions lived in past Slack threads, internal Notion runbooks, and the memory of a handful of senior support engineers, none of it searchable at the moment an agent actually needed it. New hires took months to ramp, and experienced agents spent a large share of their day re-answering questions that had already been answered dozens of times before.
The client wanted to use AI to close the gap, but they were wary of "AI support bots" that hallucinate, their product is technical, their customers are enterprise accounts, and a wrong answer sent automatically would cost more trust than the time it saved.
How Did StepTo Build the RAG Support System?
StepTo proposed a retrieval-augmented generation (RAG) system rather than a general-purpose chatbot: the LLM would only ever answer from content it retrieved from the client's own product docs, resolved tickets, and internal runbooks, never from open-ended generation. That grounding-and-guardrails approach is how StepTo builds assistants generally, described on our chatbot and LLM assistant development page.
Incoming tickets are classified and matched against a pgvector store holding embedded product documentation, 14 months of resolved tickets, and internal runbooks, refreshed nightly. A triage layer drafts a first-response answer with a confidence score; high-confidence answers to routine tier-1 questions resolve automatically, mid-confidence drafts go into a human review queue built directly into the client's existing Zendesk workflow, while low-confidence or ambiguous tickets route straight to a human agent with no draft at all. No AI draft on a ticket routed to an agent reaches a customer without that agent approving, editing, or rejecting it.
Knowledge Layer
- pgvector store
- Ingestion pipeline (Python)
- Metadata tagging
- Nightly sync jobs
Retrieval & Generation
- LangChain
- OpenAI API (GPT-4-class)
- Confidence scoring
- FastAPI backend
Integration & Review
- Zendesk API
- Human review queue
- AWS (Lambda, RDS)
- Redis cache
How Long Did the Project Take?
- Weeks 1–2
Discovery & Knowledge Audit
StepTo audited the existing docs, Notion pages, Slack threads, and 14 months of resolved tickets to establish ground truth for what the AI would be trained to retrieve.
- Weeks 3–7
Ingestion & Vector Store
StepTo built the ingestion pipeline, chunking, embedding, and metadata tagging, and stood up a pgvector store synced nightly with product docs, runbooks, and resolved tickets.
- Weeks 6–12
Triage & Drafting Layer
StepTo built the LLM-backed triage layer that classifies incoming tickets, retrieves relevant context, and drafts a first-response answer with a confidence score.
- Weeks 11–15
Human-in-the-Loop Review
StepTo integrated a review queue directly into Zendesk so every AI draft on a ticket routed to an agent is approved, edited, or rejected by that agent before it reaches a customer.
- Weeks 15–17
Pilot & Full Rollout
StepTo piloted on 20% of ticket volume, tuned confidence thresholds against real agent overrides, then rolled out to the full support queue.
What Do Companies Ask About AI Support Automation?
How do you prevent the AI from giving customers wrong answers?
StepTo grounds every response the AI drafts in retrieved passages from the vector store, not free-form generation. The system can only answer from what it actually found in your docs and past tickets. Every draft also carries a confidence score: high-confidence answers to routine tier-1 questions resolve automatically, mid-confidence drafts wait in the review queue for an agent to approve, and low-confidence tickets skip the draft entirely and route straight to an agent.
How long until this pays for itself?
StepTo's system paid back its build cost within the first quarter after full rollout. For this client, the math was straightforward: a 45% reduction in average handle time across tickets still touched by a human, plus 38% of tier-1 volume needing no human time at all, measured against the fully loaded cost of the support team's time.
What if our knowledge base is messy or outdated?
Knowledge bases almost always are, and the discovery phase exists for exactly this reason. StepTo audited what was accurate, flagged what was stale, and used 14 months of resolved tickets (not just the official docs) as a second source of ground truth, since the real answers often lived in past agent responses rather than the wiki. The system also flags recurring low-confidence topics post-launch, which becomes a live signal for where the documentation still has gaps.
Does this replace our support team?
No. StepTo's system removes the repetitive tier-1 volume so the existing team can spend their time on the tickets that actually need a human: escalations, edge cases, and anything emotionally or technically complex. Headcount on the support team was unchanged; effective capacity went up.
Related insights from the StepTo blog
- Who Actually Builds RAG Systems? A Buyer's Guide to Evaluating a RAG Development Partner
Most vendors selling a 'RAG system' have only built the easy part, a chat interface wrapped around a model call. A buyer's framework for what to ask before you sign anything.
- Your Certificates Expire Every 64 Days From February. Is Anything Actually Renewing Them?
Let's Encrypt will issue 64-day certificates by default from February 10, 2027, and the industry-wide maximum falls to 100 days in March 2027 and 47 days in 2029. Certificate renewal used to be a yearly chore. It is becoming a continuous system that has to work without anyone watching. Here is what breaks, where certificates hide, and how to build renewal that holds up.
- The Agent With a Mouse: Automating the Enterprise Systems That Never Got an API
Computer-use agents now beat the human baseline on OSWorld. That says almost nothing about whether one should be driving your ERP, and the gap between the two is where the money goes.
Let's talk about your project
Tell us what you're building and we'll get back to you within one business day with a no-obligation assessment.
Office hours
Send us a message
StepTo's Belgrade team replies within one business day.
