A voice orb that actually controls the site: the architecture of an assistant, not a “talking FAQ”
Briefly about the main thing (BLUF)
A talking widget on a website is not an assistant, but a toy: it talks but does nothing. Our voice orb really runs the site. Inside are three layers of architecture, three rakes that we stepped on, the economy of dialogue and a way to put the same orb on your website.
Voice assistants on websites usually end up in one of two ways: either it’s a chatbot with speech recognition attached, or it’s an FAQ voiceover. We are in NeuralSpace went from the inverse task - to make a voice the main way to manage the interface, and not a superstructure over the text. Below is an analysis of the architecture and decisions that had to be made.
Task
The user opens a complex generation page − video, pictures, music — where there is a choice of model, loading a reference, a request, parameters. The newcomer is lost. People close the classic onboarding (tour with arrows) at the second step. We wanted it to be possible to simply say with a voice: “I want to bring this photo to life,” and the assistant himself highlighted the desired button, explained it and brought it to the result.
Key difference from a chatbot: orb sees the page state And acts on her, rather than responding with the text “press the X button somewhere there.”
Architecture: three layers
- Vocal tract. Streaming speech recognition → model → response synthesis. The requirement is low latency and the ability interrupt (barge-in): the user starts talking - the assistant falls silent. Without this, the dialogue feels like a walkie-talkie rather than a conversation.
- Brain. We deliberately decoupled the “personality” of the assistant from a specific language model: the model is configured on the server side, so it can be changed to suit the task and cost without re-developing the front end. The assistant receives not only the user's response, but also machine-readable snapshot of the current page: what elements are there, what buttons, what is already selected.
- Executor on the client. The model returns not only text, but also actions: highlight element, scroll to block, explain field. The client executes them on top of the real DOM.
Three rakes we stepped on
The most interesting thing is not the happy path, but the failures. We analyzed the logs of real dialogues and came up with the root rules.
1. Hallucination of possibilities. The assistant suggested models that were not on the current page, or confused the model for pictures with the model for video. The user is rightly angry: “there is no such model.” Fix - don’t rely on the model’s “knowledge of the world”: the assistant can name and switch only what is actually in the snapshot of the current page. This is a classic grounding problem - the model must be strictly tied to the state of the interface, otherwise it will confidently lie.
2. The wrong button is illuminated. They ask you to show “upload a photo” - “Generate” is highlighted. The reason is the fuzzy “intent → element” mapping. Solution: elements receive semantic anchors, and the assistant must highlight exactly that one, about which he speaks, and not neighboring in meaning.
3. Obsessiveness. The model has already been selected - the assistant still tries to change it; the user asks “just write a request” - and he argues. Rule: if the state is already suitable or the user explicitly asks not to touch - do not act or argue, immediately do what is asked. Less initiative, more execution.
The conclusion we would give to anyone building an agent on top of an interface is: 90% of quality is not a model, but grounding and discipline of action. The model must see the exact state and is not allowed to go beyond it.
Economics of dialogue
Voice is more expensive than text, so Orb has a free window, then payment is made upon conversation. Billing is tied to the duration of the voice interaction, not the number of “messages.”
The same orb is on your website
We customized the orb for ourselves, but the task is universal: any site with a non-trivial interface (configurator, personal account, complex form, store with filters) benefits from a voice guide that sees the page and acts on it. Therefore, we put it into an embedded widget - how it is installed and what it can do is discussed separately: voice agent for any website.
If you are doing something similar, it is interesting to compare the approaches to grounding and barge-in. You can try orb live at neuralspace.pro.
Frequently asked questions (FAQ)
Question: How to get the best result from a neural network?
Answer: Use detailed prompts (descriptions) in English, set the style and details of the scene.
Question: Can these materials be used for commercial purposes?
Answer: Yes, the generated content is entirely yours.