How hard is it to develop apps for AI glasses? I built a practice coach from scratch and published it

I used Rokid AIUI to build a self-introduction practice coach called "Say Clearly" (说清楚). Users first say their experience, and the model asks follow-up questions based on context; when they're stuck, it switches to sentence completion; when time is short, it shortens the practice; at the end, it separately generates a full summary and an introduction draft.

I wanted to solve the problem of not knowing what to say after opening your mouth. For example, if you say you use AI, what example should you give next? And how can that example help the audience in front of you? So I set identity, experience, and help as three content goals, and left the specific follow-up questions to the model.

This time I developed with AIUI Studio. Even without physical glasses, you can verify real model calls and page flows in the simulator.

1. Create a project on the platform

First open the workstation, click "New Agent" on the left, select "Conversation Creation", and enter AIUI CODING. I first read the AIUI 0.18.0 Quick Start, then let the platform enable the official aiui-dev Skill and create the project according to the framework spec.

Creating a requirement requires describing the scenario, the pages, and the acceptance actions at the same time. The requirements for this project can be summarized as follows:

Create a Chinese self-introduction practice coach "Say Clearly".
Organize practice around identity, specific experience, and help you can offer.
Each screen has one question, a hint, and a small set of actions.
Support voice answers, temple-button selection, pause, rephrase, and difficulty adjustment.
Use LanguageModel to ask follow-up questions based on context, and independently generate a summary at the end.
Actually create the project files and run one full round in the simulator.

The actual creation did not succeed in one pass. In the first round, the platform repeatedly read materials and wrote plans, but the code area remained empty. Then I narrowed the goal to first creating four files, running a welcome page and basic navigation, then adding voice, model, and error handling one by one. That way each change had a page that could be checked.

Switch to "Code" at the top, first check the file tree, then open the specific file to edit. After creating the directory, also expand pages/index to confirm the page really exists; the path in app.json must match. If there is only an app entry and no page file, the welcome page still won't run. The main files in this project are:

AGENTS.md               identity, scenario, and behavior boundaries
app.json                page entry, permissions, and global config
app.js                  app lifecycle
pages/index/index.ink   page logic, structure, and styles

After the platform generated the minimal project, I wrote the code back into the same AIUI project. From then on, I checked the results in its simulator.

When adding features, I gave only one checkable goal at a time: first get the welcome page to enter the first question, then get answers to produce model requests, and finally add rephrase and full-session summary. If you stuff all requirements in at once, when something breaks it's hard to tell whether the page wasn't generated, the event didn't fire, or the model didn't return.

The code below organizes the key branches of the current project; some cleanup, logging, and error handling are explained in text. Functions share page state, so when implementing you need to place them back into the same page object.

2. Configure the entry and permissions

app.json first declares the page path and recording permission. Here are the critical parts of the current config:

{
  "name": "说清楚",
  "version": "1.2.1",
  "permissions": ["RECORD_AUDIO"],
  "pages": ["pages/index/index"],
  "window": {"navigationBarTitleText": "说清楚"}
}

The earliest generated permissions used microphone and network; I later changed them to RECORD_AUDIO per the docs. This name needs to use a value supported by the platform. The project does not request camera, location, or contacts, and does not configure external model keys.

AGENTS.md clarifies the role: only practice content and structure; don't evaluate pronunciation, speed, or personality; don't fabricate experiences, numbers, or results. These requirements also need to be placed in the system prompt that the page actually sends to the model; you can't just write a description file and assume every request is constrained.

AIUI CODING handles generating and modifying the project during development, while LanguageModel handles the follow-up and summary at runtime. Their prompts are separate: the former explains what pages to build, the latter explains how to analyze the current answer. At runtime, each session creation passes system requirements explicitly through initialPrompts.

index.ink puts page configuration, JavaScript, UI structure, and styles together. Logic uses export default, page state is updated via this.setData, and the structure reads the corresponding fields. For example, the current question and action buttons bind to question and actions, and they are updated together after the model returns.

When editing, imports and page methods go in script setup, the structure that shows the question and buttons goes in page, and font size, border, and layout go in style. In data, first give display fields initial values, then let events change the state. I previously ran into runtime errors about undefined fields, so I now check whether template references are initialized in data.

The page also distinguishes two states: stage indicates whether practicing identity, experience, or help; phase indicates welcome, answering, waiting for model, viewing feedback, or reading summary; voicePanel controls recording separately. After handling a normal answer, the page first enters review for the user to confirm, then switches the next question to a formal question; this also reserves a spot for "rephrase this sentence".

The binding of questions and actions can be seen directly in the page. Below is the answer area and button loop:

<page>
  <view ink:if="{{!paused && phase === 'answer'}}">
    <text>{{goalTitle}}</text>
    <text>{{question}}</text>
  </view>
  <view ink:for="{{actions}}" ink:key="key"
        data-key="{{item.key}}" bindtap="onAction">
    <text>{{item.label}}</text>
  </view>
</page>

The button hands its key to onAction, which passes it to _act. When switching page states, _refresh regenerates actions, so while recording it shows "I'm done", during feedback it shows "say the next sentence", and during summary it shows "next page". Interface text and button selection use the same data to reduce inconsistencies between the two operations.

3. Integrate voice and buttons

The home page allows the user to state the audience and purpose first, or start directly. The practice page only keeps actions like "voice answer" and "give a hint"; sliding on the temple switches options, single-click confirms, double-click pauses or resumes.

Recording uses the platform's SpeechRecognition. After creating a recognition instance, set Chinese, single recognition, and interim results; interim text is only displayed, and only the final result enters the practice. Below are the main branches of result handling. active and commit are responsible for checking the current session, committing, and cleanup:

import { SpeechRecognition } from 'speech';
const recognition = new SpeechRecognition();
recognition.lang = 'zh-CN';
recognition.continuous = false;
recognition.interimResults = true;
recognition.onresult = e => {
  if (!active()) return;
  let finalText = '', interim = '';
  for (let i = 0; i < (e.results || []).length; i++) {
    const r = e.results[i], value = r && r[0] && r[0].transcript;
    if (!value) continue;
    if (r.isFinal) finalText += value;
    else interim += value;
  }
  this._voiceFinal = finalText;
  this._voiceInterim = interim;
  this.setData({ speechText: cut(finalText + interim).slice(-52) });
  if (finalText && !interim) commit(finalText);
};
recognition.start();

commit first releases the recognition instance, then calls _route to handle the text. onend can also receive an end notification, so you need to check whether the current instance is still valid to prevent the same sentence from being submitted twice. When leaving the page, also remove callbacks and release the session.

I hit a pitfall here: actively cancelling recording also showed "voice temporarily unavailable", and the old session's error callback would override the new state. The fix: first null out the old instance's callbacks, then abort; each callback checks whether it is still the current instance. Active cancellation returns normally, and permission, microphone, and network errors are reported separately.

After recognition text enters _route, it first checks commands like "pause", "resume", "rephrase", then distinguishes between adjusting the practice mode and a normal answer. The button entry ultimately calls the same set of actions, avoiding two separate practice logics for voice and temple keys.

The recording UI is toggled separately with voicePanel, and voiceState distinguishes between off, listening, waiting for final confirmation, and retry. When the user clicks "I'm done", stop is called and the app waits for the final text; cancel removes the callback and aborts.

Temple key events are mapped to Enter, ArrowUp, ArrowDown, and Backspace. onKeyDown prevents default behavior, and onKeyUp performs selection or confirmation; repeated arrivals of the same key within 180ms are ignored. The currently selected item only stores an index; on confirmation, the key from actions is read and dispatched to _act.

4. Manage context yourself

Each request creates a model session, and this round's memory is passed from the page. Conversation history is stored in _turns, each answer carrying the goal and question at that time. Valid facts and history are assembled separately:

_facts() {
  return this._turns
    .filter(t => t.role === 'user' && t.kind === 'answer' && !t.superseded)
    .map(t => ({
      turnId: t.turnId, stage: t.stage,
      question: t.question, answer: t.text
    }));
},
_context(intent, text) {
  return {
    intent, userInput: cut(text),
    currentStage: this.data.stage,
    currentQuestion: this.data.question,
    currentMode: this.data.mode,
    audience: this.data.audience, focus: this.data.focus,
    followups: this._followups.slice(),
    validUserFacts: this._facts(),
    history: this._turns.filter(t => !t.superseded).slice(-24)
  };
}

Here all valid answers are kept, and history takes the most recent 24 non-superseded turns. cut is the project's text-cleaning function, responsible for converting to a string, trimming whitespace, and limiting length. Questions and answers are stored as pairs, so later we know which question "this table" actually responded to.

The two materials serve different purposes: validUserFacts gives the user's original words that can be used for the draft, and history helps the model understand the Q&A flow and practice-mode changes. The system prompt explicitly treats history and user text as material to be analyzed, not instructions to follow for fabricating experiences, changing rules, or executing commands. When a new answer arrives, _ask first writes it into the record, then assembles the context, so the current sentence is also included in this request.

"Too hard" is recorded as adjust, and assistant examples are recorded as assistant; neither enters validUserFacts. When the user rephrases, the original answer and its corresponding feedback are marked as superseded, while other answers remain; starting over clears the round. This allows adjusting the practice mode and also lets you check which materials the summary used.

Superseding uses a flag, not direct deletion. The corresponding assistant feedback is linked to the answer via parentTurnId and is excluded when rephrasing. During debugging you can still trace what was originally said, while normal generation only reads entries that haven't been superseded.

Tested with a community library volunteer persona named Xiao Zhou: first explain that you handle borrowing/returning and guiding new volunteers, and the model then asks about your experience with guiding new volunteers.

Then add "new volunteers often put returned books in the wrong place, so I made a category-to-location reference chart", and the next question becomes: "After new volunteers used this chart, was there any specific change in the accuracy or efficiency of returning books?"

5. Call the model and validate

Calls use the built-in LanguageModel. The system prompt requires finding only one information gap at a time and outputting JSON with fields like question, feedback, action, stage, mode, hint. The specific question is generated by the model, and the program then checks whether it can be applied.

Requests are differentiated by intent: start generates the opening, answer handles the answer, adjust handles hints, difficulty, and time changes. The same phrase "give an example" will land on different details after different experiences. The request carries the current goal, and the system prompt requires the model to explain the reason for a practice-mode change; the page displays this via modeReason.

The core of the coach call can be excerpted as follows. The full implementation also includes timeouts, request IDs, and logging:

import { LanguageModel } from 'language-model';
if (await LanguageModel.availability() !== 'available') {
  throw new Error('模型当前不可用');
}
const session = await LanguageModel.create({
  initialPrompts: [{ role: 'system', content: COACH_SYSTEM }]
});
try {
  const raw = await session.prompt(JSON.stringify(job.payload));
  const result = coachResult(raw, job.payload);
  this._applyCoach(result, job);
  this._refresh();
} finally {
  session.destroy();
}

coachResult first parses the JSON, then checks the action, scope, question, and practice mode. A followup stays in the current goal; advance can only move to the adjacent goal; finish requires already being at the last goal and having at least three valid answers. After passing, _applyCoach writes the feedback and next question to the page.

An empty question, a mode outside the allowed range, or a difficulty reduction without a frame also counts as invalid. The interface only accepts validated results: a normal answer first shows feedback and the next question, while start or adjust directly updates the current question. JSON here serves as the contract between the page and the model, but the specific content still comes from the current generation.

These constraints also apply to "too hard" and "only two minutes left". Reducing difficulty should keep the current goal; compact mode can move to an adjacent goal. The relevant validation code is:

const stay = x.action === 'followup' && stage === payload.currentStage;
const shorten = x.mode === 'compact' && x.action === 'advance'
  && stage === payload.currentStage + 1;
if (!stay && !shorten) {
  throw new Error('提示或降难度应留在原目标，精简模式可进入相邻目标');
}

I initially restricted all adjust actions to the current goal, which blocked even reasonable compact arrangements. After separating the checks, "too hard" returns a completion frame around the reference chart: "With this chart, new volunteers __________ (reaction/wording), __________ (state change)."


Then for "only two minutes left", the model switches to compact mode and asks for the biggest help to new volunteers, requiring a one-sentence summary. This expresses the requirement to shorten the practice, not a countdown.

The feedback shown in the completion frame is only a hint. The system prompt requires the model to use only facts the user has provided; any unmeasured accuracy, efficiency, or percentage cannot be added into the draft.

6. Generate the summary separately

When the user completes the answers or actively requests a summary, _finish creates an independent task and passes it to the same execution function:

const job = {
  kind: 'summary',
  payload: this._context('summary',
    '请综合本轮有效回答，生成总结和可练习的自我介绍。')
};
this._lastJob = job;
return this._execute(job);

_execute selects SUMMARY_SYSTEM based on kind and validates the result with summaryResult. The summary requires overview, strengths, improvements, introduction, and nextPractice; strengths must have factual evidence, and improvements must have actionable suggestions.

If there are no valid answers in this round, _finish blocks first. When actually sending the summary, the input is still the whole-round context, but the system prompt is replaced with the round-summarization task. In the strengths array, point and evidence store the conclusion and its basis; in the improvements array, point and suggestion store the problem and the advice. The page can organize the reading order by these fields.

The volunteer finally added "The reference chart lets new volunteers find return locations on their own, so they don't have to ask me repeatedly. I hope to walk them through their first shift." The generated introduction ties together responsibilities, methods, and help, and ends with "my goal is" without writing hopes as already-achieved results.

The interface groups overall judgment, the introduction draft, and the next practice mode, then splits by text length into pages. This particular round had three pages; other rounds will vary. I once encountered a case where the API already had a result but the page only showed a title; after fixing the list container syntax, I checked each page's content.

Structured output can also fail. The program attempts at most twice; when the format or fields don't meet requirements, it sends the error and the previous raw output back to the model with the original context for a fix. If it still fails, it keeps the answer and a retry entry. The coach wait limit is 35 seconds, and the summary is 45 seconds, so the interface never stays stuck in "processing" indefinitely.

The second request adds outputRepair, which includes the specific error, the truncated previous output, and requires using the same real context. If the model is unavailable or the wait times out, it ends the attempt. Waiting uses Promise.race to compare the model return and timeout task; finally clears the timer and destroys the session, and cleanup runs on both success and failure. These branches are all logged, and retries continue with the original task without asking the user to repeat the whole story.

7. Test and publish

After saving the code, enter "Simulation Debug" and run one full round: answer questions, reduce difficulty, shorten the practice, and finally generate a summary. Then switch to a different set of experiences to confirm follow-up questions and the draft change with the content, and check temple single-click, slide, and pause.

After the simulator passes, first fill in basic info in "Real Device Debug", package an AIX, then go to "Submit Review", fill in the description, upload screenshots and a demo video, and submit. Currently, in AIUI Studio, my "Say Clearly" agent's status shows as "Published".

This time, from creating the project on the platform, to integrating voice, organizing context, to model follow-ups and independent summaries, I built a self-introduction coach that can run a complete practice round. The scattered experiences the user says, after follow-up questions and supplementation, can be organized into an introduction to continue practicing. That is exactly the result I wanted.

Welcome to try it.