We Built an AI Gatekeeper for the Phone — and Deliberately Have Not Switched It On

Why can't a company owner simply ignore an unknown number?

Most people can afford to ignore an unknown number. A company owner cannot. A domestic number might be a delivery you actually need delivered. A foreign number might be a new client. In the overwhelming majority of cases it is neither. But the minority is the part you cannot miss, and there is no way to tell which is which without picking up.

“So I always answer unknown numbers. A domestic number might be a delivery you actually need delivered, a foreign one might be a new client. In around 99% of cases the call turns out to be genuinely low value. But as a company owner I do not have the privilege of missing the other 1%.

That 99/1 split is the entire argument for an AI gatekeeper. Something that answers first, works out what the caller wants, and only then decides whether a human is required. This article describes the AI call screening service for executives we built to do exactly that: the architecture, the five routes a call can take, and the reason it is not yet answering the phone.

That last point is not a footnote. The assistant is an internal project and it is not in production. The showcases published so far in this series describe systems that run every day. This one describes a system that is finished enough to demonstrate and deliberately switched off. The failure mode of a gatekeeper – screening out the one caller who mattered – is the kind of error you may never find out about. So what follows includes the bar it has to clear before it goes live.

Is the real cost of a low-value call the minutes or the interruption?

The intuitive way to justify call screening is to add up the wasted minutes. That turns out to be the wrong measure. Time was the cost years ago, when it still seemed reasonable to answer a market research survey that ran for twenty minutes, half of whose questions I did not understand. I do not do that any more. The modern cost is the interruption.

The canonical example is the delivery driver who calls during an important meeting to ask where to unload something. The call is genuinely useful to the company, because somebody does need to explain the exact location. It is also genuinely useless to the person receiving it. And the exit is expensive: once you have answered and discovered it is a delivery, getting rid of the caller quickly means finding and dictating a colleague's number mid-meeting.

“More than once a delivery driver has called my personal number during a very important meeting to ask where to unload something. And you still want the company goods delivered, so somebody should explain the exact location. Just not me.

An answering machine does not solve this, because it drops the useful calls along with the useless ones. What you need is something that handles the delivery driver properly – answers, understands, gives the location – and never surfaces the call at all.

Where does an AI phone gatekeeper send a call? Five destinations from one number

So what does the phone become? A front end onto systems that already exist. A call arrives on a Twilio-hosted number, the assistant answers, classifies the caller's intent, and routes the call to one of five destinations.

Whitelisted callers go straight through, unscreened

Friends, family and key clients are returned to the owner unconditionally, with no screening at all. The whitelist is currently a static list, matched on the caller's number and administered by hand. The next step is a data entry form or synchronisation with a phonebook, so nobody has to edit a JSON file directly. A later step is integration with the CRM, where the infrastructure already exists because VIS Solutions runs HubSpot.

Cold calls, surveys and delivery instructions are handled and closed by the assistant

These are the calls the assistant exists for. It answers them, deals with them, and closes them. Nothing reaches the owner in real time.

Employee admin questions are pointed at the corporate AI assistant

This is the route where the honest answer is the most useful one. The corporate assistant described earlier in this series – the one that handles HR inquiries, travel expense reporting and similar administrative work – does not have a phone channel yet. It has a chat interface. VIS's corporate AI assistant draws the same line between what the model decides and what code decides. So today the personal assistant does not transfer the call or drop a ticket into the corporate assistant's backend. It simply tells the employee, with short instructions, to use the corporate assistant. A phone channel for the corporate assistant is expected within roughly a year, and at that point this becomes a real handoff.

Client support requests are forwarded to support, where a ticket is created from the call

A caller with a support request is forwarded directly to the support number. There a ticket is created from the phone call, exactly as it would be in a human-run support centre. VIS's support automation has its own phone channel and will be the subject of its own series of articles, so the internals of the ticket creation step are out of scope here.

High-value and unclassifiable calls are escalated to a human

New sales conversations, key clients who are not on the whitelist, and any call the assistant cannot confidently classify are escalated to the owner. The unclassifiable case defaults to escalation on a simple rationale: without the assistant, that call would have been answered anyway.

Four of the five routes have been exercised; the support forward has not

Of these five, four have been exercised in testing. The client support forward is the exception, and for a mundane reason: the prototype was developed on the same Twilio number that support now uses, so the forward has never been tried end to end. Buying a second number fixes it. It is not the reason the system is switched off.

How is the voice agent built, and what is the model actually allowed to decide?

The number is hosted on Twilio. The code runs on Railway. It started on Vercel – the hosting choice was originally an AI recommendation. Railway turned out to be the better fit for this kind of serverless work.

The division of labour inside the call is the important architectural decision. The language model does two things: it does the talking, and it classifies the caller's intent. Everything downstream of that classification is deterministic and rule-based. There is no model deciding, mid-call, whether to dial a support number.

“The model does the talking and the intent classification. Once intent is established, the rest is deterministic and rule-based.

The model itself is not yet fixed. The choice is between a fast, small model and a more capable, slower one, and it will be settled by experimentation before go-live rather than by preference. On a live phone call, latency and classification quality pull in opposite directions.

Why the realtime streaming voice path was parked for Croatian

On the voice layer, two approaches were built. A realtime streaming pipeline was implemented and then parked, because of latency problems when speaking Croatian. The assistant currently uses simpler turn-taking: the caller speaks, the speech is transcribed, and the assistant responds. Text to speech is ElevenLabs. During testing the voice was a clone of the owner's own voice, and the intention is to keep it in production. People expect to hear his voice when they call his number. And callers will be told they are speaking to an AI.

Callers are told they are speaking to an AI, and recording is settled before go-live

“We should tell them it is an AI. That is the decent thing to do, if it is not outright imposed by regulation. The voice? Mine, at first – people expect to hear my voice on my number.

On recording and GDPR the position is equally plain: at production, recording is either turned off or the caller is informed that the call is being recorded.

What happens when an AI gatekeeper screens the wrong call?

Screening is a classification problem with two very different error costs. A false positive – forwarding a cold caller – wastes a few minutes. A false negative – screening out a genuine prospect – loses a deal, and does so invisibly. The two errors are not symmetric, and no amount of prompt tuning makes them symmetric.

Two mechanisms handle it. The first is the default: when in doubt, escalate. The second is visibility. Every call is logged, and a daily summary email lists the calls with number, caller, time and a short summary of what happened. Those digests are already being received and read during testing. They are what makes screening acceptable rather than sinister – nothing is dropped silently, and a call that was handled badly shows up the same day.

Errors did occur in testing, as they do in every build of this kind. So the realistic expectation is stated rather than hidden: even with system prompt tuning, some high-value calls will occasionally be missed and some low-value calls will occasionally be forwarded. Cold callers are, after all, actively optimising to sound like the 1%.

Why is the assistant deliberately not in production yet?

There are two reasons, and only one of them counts. The first is the shared Twilio number, which is a trivial fix. The second is the real blocker: the owner wants everything double-checked and tested before a machine is allowed to handle his personal number.

Concretely, that means running a set of real calls that exercises every scenario in the routing table before go-live. Beyond that, there is a plan to hand the codebase to a coding agent – Claude Code – to implement machine-to-machine tests, smoke tests and further development iterations. The expectation is explicitly not a rewrite. The project was started in 2025 and hand-coded before VIS's other agentic solutions, and the aim is for it to stay familiar to the people who first wrote it. Nor is there a finish line. As with every other production solution here, development is not expected to ever be done.

What is missing from that bar, honestly, is a number. There is no target pass rate and no defined shadow-running period yet. The criterion today is a full scenario sweep plus an automated test suite, judged by the person whose phone it is.

What can this AI phone assistant not do yet?

Beyond not being live, the limits of the current design are specific:

• It cannot call anyone back. Callback is not implemented.
• It does not take messages. A call it cannot classify is forwarded, not recorded for later.
• It speaks Croatian and English. Other languages have not been tried.
• The whitelist is maintained by hand, with no phonebook or CRM synchronisation yet.
• The client support forward has never been executed end to end.

How well does speech recognition cope with Croatian calls?

The most entertaining part of testing was Croatian speech to text. Speech recognition is noticeably weaker in Croatian than in English, and personal names suffer worst of all. Test calls were role-played as inbound enquiries from CEOs of some of the largest companies in Croatia, and the transcripts mangled their names into something ridiculous with some regularity. It did not appear to affect the routing. Intent classification did not seem to depend on getting the name right, wrong or otherwise.

Do enterprise executives have this problem too?

Today this is an internal pet project on one person's phone, and it should be described as nothing more. Whether it becomes a commercial product is an open question. The argument that it might is simple arithmetic about attention: if the owner of an SME receives this volume of low-value calls, an enterprise executive is unlikely to receive fewer. Separating the wheat from the chaff at the phone layer is worth at least as much there.

The architectural point generalises further than the executive gatekeeper use case. The interesting part of this build is not the voicebot; voicebots are commodity. It is that the phone became a routing surface onto systems that already exist – the corporate assistant, support automation, and the human of last resort. The language model does only the two things language models are good at, talking and classifying.

What the payoff looks like on the day it is switched on

“When it goes to production, and I see a cold caller handled without ever bothering me, that will be one of the nicer moments. Compensation for hundreds of cold calls where the caller would not quit until I hung up.

That is the whole promise of the thing, and it is also why it is still switched off. A gatekeeper that handles the cold caller is a small pleasure. A gatekeeper that hangs up on a client is an expensive mistake. The second has to be ruled out before the first is worth having.

AI voice agent with intent routing compared with voicemail, an IVR menu and a human answering service

You almost certainly have callers you need to hear from and callers that kill your focus, with no reliable way to tell which is which before you pick up. The phone layer is just the beginning – the same intent-routing pattern appears in everything we build next, and we are publishing the architecture of each system as it reaches testing, including the parts that are deliberately not in production yet. Subscribe to follow what happens when they go live.