Designing a General-Purpose Intelligent Phone Platform from Scratch: FreeSWITCH + ESL + AI in Production

Many teams building 'AI answers the phone' put the bot directly into the FreeSWITCH dialplan in the first version. The demo runs, but soon they hit problems: switching ASR vendors means changing call code, adding an extension means editing XML, calls mysteriously drop after transfer to a human, and CDRs don't match.

This post takes a different approach: design the general-purpose phone platform first, then let AI plug in as an add-on. Below are the most critical and error-prone parts.

1. Scenarios and Goals

The platform covers three scenarios: customer service inbound (bot answers first, complex issues transfer to a human), outbound callbacks/notifications (callbacks, reminders, appointment confirmations), and human-machine collaboration (bot transfers to a human with a summary, and after the agent finishes, the call returns to the bot for a satisfaction survey).

The goal is five 'can's: connect (inbound/outbound, interruptible), pluggable (switching bots or vendors requires no call-code changes), operable (configuration changes happen in the admin console, not by logging into a server), verifiable (the full chain can be tested from a browser), and scalable (a single node can grow into a cluster).

2. Layered Architecture: Let FreeSWITCH Be the Executor Only

The system is divided into five layers:

Layer	Responsibility
Access layer	Carrier trunks, SIP phones, browser softphones; add Kamailio for registration and distribution when clustering
Media layer	FreeSWITCH: signaling, media, transcoding, playback/recording, bridging
Control layer	Configuration service (xml_curl), call control engine (ESL), queue and outbound scheduling
AI access layer	AI access gateway + external bots or ASR/TTS/LLM adapters
Business layer	Admin console, agent toolbar, open API, QA reports

The core principle in one sentence: FreeSWITCH only handles signaling, media, and executing commands; configuration lives in a database, decisions live in ESL programs, and AI connects through a unified protocol.

Supporting principles: only the control engine holds ESL; when AI needs to interrupt or transfer to a human, it sends an event to the engine rather than touching channels directly. On failure, either hang up or return to waiting; no zombie channels. One call_id through the entire chain, carried from channel variables to logs, AI requests, and CDRs.

3. Database-Driven Configuration: mod_xml_curl + One-Click Publish

Editing XML and running reloadxml gets hard to manage once extensions and nodes grow. mod_xml_curl works differently: when FreeSWITCH needs configuration, it sends an HTTP request to your service to fetch it.

<param name="gateway-url" value="http://10.0.0.11:8080/fs/xml"
       bindings="directory|dialplan|configuration"/>

The configuration service dispatches by the section in the request: directory returns extensions, dialplan returns the dialplan, and configuration further distinguishes gateways, ACLs, etc. by key_value. A few key points:

• If nothing is found, return <result status="not found"/>; do not return empty content.
• Extension passwords are delivered as a1-hash, never plaintext.

Static/dynamic boundary: global variables, module lists, profile skeletons, and ESL configuration stay in static XML; extensions, gateways, routes, and ACLs live in the database. Static XML should also keep a fallback dialplan: if the configuration service is down, play system busy instead of leaving the customer in silence.

Once in the database, there are two kinds of changes:

• Routes and extensions: take effect on save. They are fetched in real time on incoming call or registration, so the next call or registration uses the new config. If cacheable is enabled, the program automatically runs xml_flush_cache after saving.
• Gateways and ACLs: require "publish". Gateways are registered by FreeSWITCH to carriers, while extensions are registered by phones to FreeSWITCH, so changes don't take effect automatically.

Clicking publish validates first, then runs commands over ESL: for a new gateway, sofia profile external rescan; for modification or deletion, killgw first, then rescan; for ACLs, reloadacl. Then check whether the gateway is REGED; mark it "published" on success, or report an error. For larger multi-node deployments, consider gradual rollout.

4. Call Control: Unified Park, ESL Takes Over

All calls park first

The dialplan no longer encodes complex routing; it just returns a unified entry point:

<action application="set" data="vr_call_id=${uuid}"/>
<action application="set" data="park_timeout=20"/>
<action application="park"/>

The control engine, connected through ESL (inbound mode), receives CHANNEL_PARK and decides whether to send the call to a bot, IVR, or human based on number, schedule, and agent state. Inbound, outbound, and transfers share the same state machine; park_timeout prevents a channel from hanging forever if the engine fails.

Events are dispatched to fixed workers per call

If the ANSWER, PLAYBACK_STOP, and HANGUP events of one call are processed concurrently, ordering is easily broken. The reliable approach is: fix N workers, hash by call ID, so the same call is always processed serially in the same worker, while different calls run in parallel.

key = ev.get("variable_vr_call_id") or ev["Unique-ID"]
await queues[hash(key) % N].put(ev)   # bounded queue; if full, backpressure or drop non-critical events

Using the whole call's call_id as the key (falling back to UUID when absent) means events from legs A and B don't race each other.

Command combinations worth knowing

• Outbound: the application pre-generates origination_uuid, writes cache first, then sends bgapi originate ... &park(); after answer, the program decides the next step.
• Transfer to human: set park_after_bridge=true before bridging; when one side hangs up, the other side returns to park instead of being dropped with it.
• Unbridge and return to bot: uuid_transfer <uuid> -both 'set:park_after_bridge=true,park:' inline.

Redis state needs a TTL

Call state stored in Redis must have an expiration time. If ESL disconnects and misses hangup events, the lack of TTL leaves permanent dirty data. After ESL reconnects, immediately reconcile with show channels against Redis: for entries in Redis but not in FreeSWITCH, write an "abnormal end" CDR, then clean up. Reconnect uses exponential backoff (1s, 2s... up to 30s), not a once-per-minute poll.

5. Pluggable AI Access: One Protocol for Many Vendors

How media gets out

The FreeSWITCH-to-AI segment uses mod_audio_stream: uuid_audio_stream <uuid> start <ws address> mono 8k <metadata> pushes call audio as PCM frames over WebSocket. One caveat: in the community edition, the downstream direction is "the server returns base64 audio, the module writes it to a file, then plays it"; true bidirectional streaming is a commercial feature. So for an MVP, return utterances one by one and play them one by one, and keep the first utterance as short as possible.

Unified protocol + two integration modes

The platform provides an AI access gateway that converts FreeSWITCH media into a unified WebSocket protocol:

Direction	Message	Purpose
Platform → bot	session.start / session.end	Session start/end, with scenario and audio format
Platform → bot	binary PCM, dtmf, playback.finished	User voice, keypresses, playback progress
Bot → platform	response.audio, transcript.final	Audio to play, recognized text
Bot → platform	response.cancel	Interrupt: platform immediately runs uuid_break
Bot → platform	action.transfer / action.hangup	Transfer to human (with summary) / hang up

Integrators have two options:

• Whole-bot integration: a bot with its own VAD, ASR, LLM, and TTS implements this protocol and can go onto phone lines.
• Component integration: if you don't want to build a bot, the platform's built-in orchestration calls ASR, TTS, and LLM adapters; switching vendors means swapping adapters.

My open-source voice-robot is an example of whole-bot integration. It was originally a browser voice robot (Tencent ASR, Volcano Ark, Volcano streaming TTS), already with streaming output and cancel + generation_id interruption. To connect it to the phone, only a protocol conversion layer is needed: the server adds the VAD that used to run in the browser, automatically commits each turn, and maps generation_id to response_id.

Interruption and latency

The key to interruption: when user speech lasts longer than a threshold (e.g. 200-400ms, needs tuning) or non-empty text is recognized, the control engine runs uuid_break <uuid> all and cancels the current LLM and TTS turn. Backchannels like uh-huh and okay can be configured not to interrupt, and compliance statements can be marked as non-interruptible.

On latency, end-of-speech detection, ASR, LLM first token, and TTS first packet together add up to a rough estimate of 1-3 seconds; this needs real-world measurement. Compress it with end-to-end streaming, short first utterances, and pre-synthesized fixed scripts. Phone audio is 8k narrowband, so prefer an 8k phone ASR model.

6. Browser WebSocket Softphone: The Entry Point for System Verification

Opening a web page to log in as an extension, dial, and answer is the easiest way to verify the whole system.

Choice: SIP over WSS + WebRTC

FreeSWITCH offers two paths: Sofia's ws/wss port (standard SIP, clients use SIP.js or JsSIP), or mod_verto (private JSON-RPC). The recommendation is the former: a standard protocol, mature clients, and it can sit behind Kamailio when clustering.

FreeSWITCH configuration points

• Certificates: wss-binding defaults to :7443, and the certificate goes in wss.pem. Without a certificate, FreeSWITCH auto-generates a self-signed one, but browsers reject it. Use a real certificate for a domain; if you only have an IP, Let's Encrypt can now issue IP certificates, but they expire in about 6 days, so automatic renewal is required.
• Public address: on cloud hosts, set ext-sip-ip and ext-rtp-ip to the public IP, otherwise the SDP contains private addresses, which shows up as one-way audio or silence. Also don't use the default private IP for domain, or registration authentication fails.
• Encryption and NAT: WebRTC mandates DTLS-SRTP; when corporate networks block UDP, you need TURN (such as coturn), and TURN credentials are dynamically generated by the backend per login.
• Codecs: browsers use OPUS, carrier trunks mostly use PCMA/PCMU, so FreeSWITCH transcodes; remember to stress-test CPU.
• Ports: open 7443 and the RTP port range; never expose 8021 (ESL) to the public internet.

Identity and two WebSockets

The browser does not store a fixed SIP password: after an agent logs into the platform, they get a temporary SIP password (stored in Redis, valid for the login session). During registration, FreeSWITCH queries via xml_curl, and the platform returns a1-hash or rejects based on login state. On logout, run flush_inbound_reg to invalidate the registration immediately.

The agent page has two WebSockets: a signaling WebSocket to FreeSWITCH for registration and calls, and a business WebSocket to the platform for agent status, incoming-call screen pop, and busy/idle. The rule is that an incoming call is assigned only when "SIP registered + business connection online + status idle" all hold, and screen pop comes before ringing.

How to verify

After the environment is set up, first log in with two browsers as 1000 and 1001 and call each other to confirm two-way audio. Then call in from an outside line and check whether the browser agent gets a screen pop before ringing. Finally, run through "bot answers → transfer to human". If all three pass, the registration, media, routing, and AI access chain is working end to end.

7. IVR and CDRs

Tree-shaped IVR: interpret execution instead of generating config files

A common IVR configuration is a tree: each node has a name, key, prompt text, and handling method (enter a submenu, transfer to agent, transfer to extension, collect digits, such as the last four digits of an ID card); * repeats, # goes back to the parent. Core fields of the node table:

ivr_node(id, flow_id, parent_id, node_name, dtmf_key, prompt_text,
         action_type, action_target, collect_cfg, timeout_sec, max_retries)

When clicking "apply IVR": validate for key conflicts and target existence, pre-synthesize prompt audio, and generate an immutable version. The recommended execution method is interpretation by the ESL program: the control engine issues play_and_get_digits node by node and moves to the next node based on keypresses. Compared with generating ivr.conf, this can directly transfer to queue, transfer to bot, record keypress traces, and publish without reload; ivr.conf can remain as a fallback menu.

One small conflict to note: in a digit-collection node, # is the end key, so you can agree that "press # without entering any digits" means return to parent.

The difference between billsec and duration

"Call duration" and "talk duration" are the most common questions in CDR lists:

• duration: from call start to end, includes ringing; it has a value even if the call was not answered.
• billsec: from answer to end; this is the billed time, and it is 0 if not answered.

One more trap: on inbound calls, as soon as the IVR or bot answers, the customer leg counts as "answered", so the customer leg's billsec includes IVR, bot dialogue, and queue time. Agent talk time should come from the agent leg's billsec. Also, one call has at least two legs; determine the primary leg before doing statistics, otherwise counts double. CDRs are written asynchronously to the database through a message queue and reconciled with native CDRs.

8. Security Pitfalls: Default Password 1234

FreeSWITCH's default config ships with 20 demo extensions, 1000-1019, all with the default_password from vars.xml; the official default is 1234. When the server has a public IP, scanners show up quickly, and a successful registration lets them make outbound calls on your line; the bill can explode overnight.

Must-do checklist:

• Change the default password before going live. After switching to xml_curl for extension delivery, delete the demo users in the static directory. Otherwise, when xml_curl returns not found, FreeSWITCH falls back to static config and the old password still works.
• ESL listens only on the internal network and has an ACL; the xml_curl interface needs authentication; don't print credentials in logs.
• Block international and high-rate number ranges by default; limit concurrent calls and daily outbound volume per extension and per tenant; alert and automatically disable on abnormal outbound calls.
• Outbound calls need user consent records, blacklists, and time-window controls. Per Ministry of Industry and Information Technology rules, call centers must retain recordings and consent records for at least 30 days; commercial marketing outbound calls are not allowed. Consult legal advice for specifics.

Final Words

This design comes down to three key choices: configuration into a database, decisions into ESL programs, AI through a unified protocol. With those in place, switching vendors, adding nodes, or changing flows doesn't require touching FreeSWITCH.

If you want to try it, take voice-robot and a FreeSWITCH instance, and start from "log in as an extension in the browser, call the bot, interrupt it by speaking". Comments are welcome on pitfalls you've hit.