跪拜 Guibai
← Back to the summary

A 0.9B Model on a 5060 Ti Just Solved the Meeting-Minutes Problem

by 雪隐_上班了 from juejin.cn/user/143341…
Sharing and aggregation are welcome, but full-text reprinting is not necessary. Please respect copyright. The circle is only this big. If you need it urgently, you can contact me for authorization.

Column Purpose: Use my RTX 5060 Ti 16G + 64GB RAM "budget battle wagon" to do something novel that justifies the electricity bill.

In previous chapters, we made the graphics card "understand human speech" (Whisper), "draw" (ComfyUI), and act as an "AI programmer" (Claude Code + Bonsai). This chapter goes for something more down-to-earth—let AI attend meetings for us (more precisely, take meeting minutes for us).

1. Meeting Minutes, the Number One Torture in the Workplace

Every company has a job not written into any JD but assigned to everyone: taking meeting minutes.

A meeting lasts an hour, organizing the minutes takes two hours. After organizing, you send them to the group chat, and no one reads them. Next week, everyone forgets what was said last time, so you meet again, take minutes again, and again no one reads them. This cycle is historically known as the "Minutes Closed Loop" — closed loop means no one can escape from it.

I decided to outsource this torture to AI. The requirements were clear:

The final result: Upload recording → take a sip of water → get a Markdown-formatted minutes document with speaker labels, complete with topics, key points, resolutions, and to-dos. This article explains how this system was built, what the core code looks like, and the VRAM landmine I stepped on along the way.

2. Selection: Why Not Whisper Anymore

I previously always used Whisper Large V3 for subtitles. It's great, but it has a fatal flaw: it doesn't know who is speaking.

Whisper only converts speech to text. Whether a sentence was said by the boss or by you—it doesn't care. It's just an emotionless dictation machine. Give it a three-hour debate recording, and it gives you a transcript where "one person talks from start to finish." Who said what? Unknown. Who argued with whom? Unknown. Who made the final decision? Unknown.

For meeting minutes, speaker diarization is a hard requirement. The traditional solution is to cobble together an ASR + independent diarization pipeline—you run Whisper to transcribe, then run pyannote to separate speakers, and finally use a script to align the outputs of the two systems. Anyone who has done this knows the pain of integrating two systems. Timestamps don't match, speaker IDs get mixed up, segmentation logic conflicts... Debugging this pipeline is more painful than the meeting itself.

Then I discovered MOSS-Transcribe-Diarize 0.9B (OpenMOSS, released July 2026, won first place in the INTERSPEECH 2026 MLC-SLM Challenge):

In official evaluations, its cpCER (speaker-attributed character error rate) beat Doubao, ElevenLabs, and Gemini 2.5 Pro. Even more impressively, on the Alimeeting dataset, Δcp = -2.69, meaning its speaker attribution labeling was more accurate than the human reference annotations.

What does this equate to? The student graded the teacher's homework and corrected it. The teacher checked against the answer key and found the student's labeling was more accurate than the reference answer—so who is the teacher now?

For the LLM organization layer, my strategy is local-first (LM Studio):

Task Model Context Reason
Meeting Minutes / Courseware Organization google/gemma-4-26b-a4b-qat 200K Easily fits the transcript of a one-hour meeting, no truncation needed
Subtitle Translation hy-mt2-7b 4K Small translation-specific model, fast and economical, doesn't stumble on sports terminology
Fallback DeepSeek API — Used only if local fails, costs money but saves the day (hasn't been needed yet)

3. Overall Architecture (Understand with One Diagram)

Recording/Video
   │
   ├─[1] ffmpeg normalization → 16kHz mono WAV        core/media.py
   │      (Standardizes various messy formats into food the model can eat)
   │
   ├─[2] MOSS transcription + speaker diarization      core/transcribe.py
   │      → [S01] 14:32 I think this requirement is very simple
   │      → [S02] 14:35 You said the same thing last time
   │
   └─[3] Local gemma organizes into structured minutes  core/meeting_minutes.py
          (LLM scheduling: core/llm_client.py)
          → # Meeting Minutes → Topic → Attendees → Discussion Points → Resolutions → To-Dos

The entire project is a Flask web application (three tabs: Subtitle Generation / Meeting Minutes / Courseware Organization), but the core is all in the core/ package. Let's dissect them one by one.

4. Core Code Dissection (Programmer Time)

4.1 MOSS Transcription Backend: Lazy-Loading Singleton

Loading the model takes several seconds, so obviously you can't reload it for every audio segment. The old rule: lazy loading + singleton — load only on the first call, then respond instantly:

# core/transcribe.py
class MossTranscriber:
    _instance = None
    _lock = threading.Lock()

    @classmethod
    def get_instance(cls) -> "MossTranscriber":
        with cls._lock:
            if cls._instance is None:
                cls._instance = cls()
            return cls._instance

    def _load_model(self):
        if self.model is not None:   # Already loaded, return directly, don't reinvent the wheel
            return
        from transformers import AutoModelForCausalLM, AutoProcessor

        self.dtype = torch.bfloat16 if self.device == "cuda" else torch.float32
        self.model = AutoModelForCausalLM.from_pretrained(
            self.model_path,
            trust_remote_code=True,          # MOSS is a custom architecture, must enable
            dtype="auto",
            attn_implementation="sdpa",      # ← Key point! Tested later in Landmine One
        ).to(dtype=self.dtype).to(self.device).eval()
        self.processor = AutoProcessor.from_pretrained(
            self.model_path, trust_remote_code=True,
        )

Design Pattern Mini-Class: This is called the Singleton pattern. Translated into human speech—only keep one cat in the entire program, and everyone uses the same one. Saves cat food, and saves VRAM.

Transcription itself uses the official toolkit in two steps: construct messages, generate:

def transcribe(self, audio_path, hotwords=None, max_new_tokens=8192, prompt=None):
    self._load_model()
    from moss_transcribe_diarize.inference_utils import (
        build_transcription_messages, generate_transcription,
    )

    # Official default prompt + optional hotwords
    final_prompt = prompt or DEFAULT_PROMPT
    if hotwords:
        final_prompt += "Hotword hints: " + ", ".join(hotwords)

    messages = build_transcription_messages(audio_path, prompt=final_prompt)
    result = generate_transcription(
        self.model, self.processor, messages,
        max_new_tokens=max_new_tokens,
        do_sample=False,               # Transcription needs determinism, no creative freedom for the model
        device=torch.device(self.device),
        dtype=self.dtype,
    )
    segments = self._parse_segments(result["text"])
    ...

The line do_sample=False is worth highlighting. Transcription is not poetry writing; it doesn't need creativity or "rephrasing." If you let the model improvise, it might transcribe "The boss said add a requirement" as "The boss said give me a raise" — one word off, a world apart.

The raw text the model outputs looks like this:

[2.80][S01] How's it going, how's it going. [4.20][3.80][S02] Because I also can't match it up... [11.70]

parse_transcript is responsible for splitting it into structured segments of {start, end, speaker, text}. I just need to filter out illegal timestamps where end <= start (the model occasionally talks in its sleep, like giving an end time earlier than the start time—a spacetime paradox, basically).

4.2 Speaker Text: The "Script" Fed to the LLM

MOSS's output is already structured, but throwing it directly at the LLM is too verbose. Compress it into a "script" format — speaker + timestamp + content, one line per entry:

# core/meeting_minutes.py
def _format_speaker_transcript(segments: list) -> str:
    lines = []
    for seg in segments:
        m, s = divmod(int(seg["start"]), 60)
        h, m = divmod(m, 60)
        ts = f"{h:02d}:{m:02d}:{s:02d}" if h else f"{m:02d}:{s:02d}"
        speaker = seg.get("speaker") or "S??"
        lines.append(f"[{speaker}] {ts} {seg['text']}")
    return "\n".join(lines)

Result:

[S01] 00:32 This requirement is very simple, I don't care how you implement it
[S02] 00:45 You said the same thing last time, and then we worked on it for a month
[S01] 01:03 That's a technical issue, let's not discuss technical details today

Looks like a script, reads like a script, and the LLM processes it like a script—who said what, and when they said it, is crystal clear.

4.3 The Soul of the Minutes: System Prompt

For the LLM to organize minutes, 90% of the skill is in the prompt. The structure is rigidly fixed, and its mouth is sealed shut:

MEETING_SYSTEM_PROMPT = """You are a professional meeting minutes assistant. I will provide a meeting transcript with speaker labels
and timestamps ([S01], [S02], etc. are anonymous speaker IDs). Please organize it into structured meeting minutes.

Output requirements (strictly follow this structure):

# Meeting Minutes
## Meeting Topic
(Summarize in one sentence; if the topic cannot be determined from the transcript, write "Unable to determine from transcript")
## Attendees
(List by speaker ID; if roles can be inferred, annotate them, e.g., "S01 (Moderator)"; if not, just list the IDs)
## Discussion Points
(Organize by topic into subsections, label the main speaker for each point, e.g.: - (S01) ……)
## Resolutions
(Clearly list the conclusions reached, item by item; if none, write "None")
## To-Do Items
(Format: - [ ] Item (Responsible: S0x, Time: if mentioned); if none, write "None")

**Writing Standards**:
- Do not fabricate any content, data, or names not present in the transcript
- Do not output any preamble, start directly from # Meeting Minutes"""

Two design details are worth mentioning:

  1. "If none, write 'None'" : Leaves no room for the model to "force something." AI's ability to fabricate resolutions is far stronger than a human's; it must be restrained—otherwise, it could invent ten items of "consensus reached" when the entire meeting was actually an argument.
  2. "If roles cannot be inferred, just list the IDs" : Prevents the model from imagining "S01 is Boss Zhang." What if they're an intern? If the name is wrong, the minutes are useless.

Real-world testing showed excellent results. I deliberately fed it a single-person baseball commentary (not a meeting at all), and the model honestly wrote:

"The transcribed content is a monologue, not involving meeting discussions or decisions, possibly an interview or commentary clip."

Resolutions: None. To-Do Items: None.

No drama queen behavior, thumbs up. You have to know, some models, even if you feed them the sound of rain, will fabricate a resolution like "Attendees unanimously agreed that it is raining today."

4.4 LLM Scheduler: Local-First, Model Assignment by Task

This is the piece of code I'm most satisfied with in the entire project. The requirements are complex:

# core/llm_client.py
TASK_PROFILES = {
    # Subtitle translation: hy-mt2-7b, 4096 context
    "translate": lambda: (config.LOCAL_LLM_MODEL_TRANSLATE,
                          config.LOCAL_LLM_MAX_INPUT_TOKENS_TRANSLATE),
    # Minutes/Courseware: gemma-4-26b, 200K context
    "reasoning": lambda: (config.LOCAL_LLM_MODEL_REASONING,
                          config.LOCAL_LLM_MAX_INPUT_TOKENS_REASONING),
}

def _backend_order(user: str, task: str = "reasoning") -> list:
    """Determine the order of backend attempts based on task type and input length"""
    local_model, local_max_tokens = TASK_PROFILES[task]()
    local = ("local", local_model, 900)
    deepseek = ("deepseek", config.DEEPSEEK_MODEL, 120)

    backends = []
    if config.LOCAL_LLM_ENABLED:
        est = estimate_tokens(user)          # Rough estimate: Chinese characters × 0.7
        if est <= local_max_tokens:
            backends.append(local)           # Fits → local first
        else:
            print(f"Input ~{est} tokens, exceeds local model limit, using DeepSeek directly")

    if config.DEEPSEEK_API_KEY:
        backends.append(deepseek)            # DeepSeek is always the last safety net
    return backends

Design Thought: Don't use the clumsy "try local first, switch on timeout" approach. Instead, calculate the input length first. If it fits, use local; if not, go directly to the cloud. Saves one futile call and 900 seconds of timeout waiting.

The fault-tolerance logic in the calling layer is also nuanced:

for name, model, timeout in _backend_order(user, task):
    client = _get_client(name)
    for i in range(max_retries):
        try:
            return client.chat.completions.create(...)
        except openai.APIConnectionError:
            break        # Connection failure = service not up, retrying is pointless, switch backend directly
        except Exception:
            time.sleep(2 ** i)   # Other errors (rate limiting, etc.) → exponential backoff retry
raise RuntimeError("All LLM backends failed")

Connection errors and other errors are handled separately:

Courseware material for one lecture is about 48K tokens. Previously, it had to go through DeepSeek (wallet bleeding). After switching to gemma with 200K context, it swallows it locally in one gulp—this is the meaning of assigning models by task: translation uses a small model (fast), reasoning uses a large model (high capacity), each doing its job, no waste.

4.5 Subtitles: Another Way to Open the Transcription Results

Using the same MOSS backend, subtitles are a breeze. Worth mentioning is the duplicate segment merging—ASR models occasionally "stutter," outputting several nearly identical subtitle lines in a row:

[0.0-1.0] Today let's talk about
[1.0-2.0] let's talk about Python
[2.0-3.0] Python decorators

Merge them using LCS similarity:

# core/srt_writer.py
def merge_duplicate_subtitles(segments, similarity_threshold=0.85):
    while i < len(segments):
        current = segments[i]
        merged_text, merged_start, merged_end = current["text"], current["start"], current["end"]
        while i + 1 < len(segments):
            next_seg = segments[i + 1]
            similarity = _calculate_similarity(merged_text, next_seg["text"])  # LCS
            if similarity >= similarity_threshold:
                merged_end = max(merged_end, next_seg["end"])  # Swallow the next one
                i += 1
                continue
            break
        ...

LCS Longest Common Subsequence: In human terms—"The weather is nice today" and "The weather is nice today, huh" share 80% of the same characters, likely a repetition, so merge them.

Speaker labeling is a toggle: [S01] dialogue is directly prepended to the subtitle text, so you can tell who's speaking when watching US/UK dramas.

Translation uses the local hy-mt2-7b, running in small batches (5 lines per batch—local models have a bad habit of merging consecutive sentences; if the batch is too large, it will arbitrarily combine three lines into one, messing up the line count). In testing, it accurately translated "walk-off home run" to "再见本垒打" — it even knows baseball terminology, better than I do.

Courseware organization is similar: batch transcription of lecture videos + PDF text extraction (PyMuPDF) + gemma fusion into Markdown courseware with LaTeX formulas. A lecture on the BHB attribution model generated 16,000 characters, all formulas correct—considering LLMs frequently stumble on mathematical formulas, this time it didn't, and I was so moved I almost gave it an award.

5. Landmine Record: The 49.55 GiB VRAM Bomb

The first test with a real recording—voice recorder's REC001.WAV, 132.9 MB, 36 minutes. Confidently clicked "Generate Meeting Minutes," ten seconds later:

CUDA out of memory. Tried to allocate 49.55 GiB.
GPU 0 has a total capacity of 15.93 GiB

49.55 GiB.

My graphics card has a total of 16G, and it demanded 49.5G in one go. This isn't just a budget overrun; this is asking my graphics card to take out a mortgage.

My mood at the time:
"You want 49.5G? I only have 16G total. How about you wait, and I'll go borrow one from next door?"
—The graphics card did not respond. It was dead.

Cause: The model's remote code defaults to full attention, and VRAM usage grows quadratically with audio length. A 36-minute audio expands into a sequence of tens of thousands of tokens, and the attention matrix launches directly into orbit. Moreover, my VRAM has to share a room with gemma 26B in LM Studio—rent was already tight, and you want to live in the presidential suite.

The fix has two layers (the foreshadowing buried in section 4.1):

# Layer 1: Switch to SDPA efficient attention at load time, killing the O(n²) VRAM allocation
model = AutoModelForCausalLM.from_pretrained(
    ..., attn_implementation="sdpa")   # One line saves tens of GB

SDPA (Scaled Dot-Product Attention) is PyTorch's efficient attention implementation, reducing VRAM usage from O(n²) to O(n). One line of code, down from 49.5G to an acceptable range.

What if it still doesn't fit?

# Layer 2: Automatically slice into 10-minute chunks, transcribe in segments, then stitch together
try:
    result = generate_transcription(...)
except torch.cuda.OutOfMemoryError:
    torch.cuda.empty_cache()
    return self._transcribe_chunked(audio_path, chunk_seconds=600, ...)

The core of chunking is one thing—timestamp offset:

for idx, chunk_path in enumerate(chunks):
    offset = idx * chunk_seconds                  # Offset by 600 seconds per chunk index
    result = generate_transcription(...)
    for seg in self._parse_segments(result["text"]):
        seg["start"] += offset
        seg["end"] += offset
        all_segments.append(seg)
    torch.cuda.empty_cache()                      # Clean VRAM fragmentation between chunks

The chunking scheme has a known side effect: speaker IDs are independent per chunk—[S01] in chunk 1 and [S01] in chunk 3 might not be the same person. It's like swapping in a new batch of anonymous extras every ten minutes; you think four people are meeting, but actually four groups are taking turns on stage.

How to get globally consistent IDs? Unload gemma from LM Studio first, freeing up VRAM to let a single inference run through. The trade-off is given to you; choose yourself.

Final result: 36-minute recording → 4 chunks → 895 segments, 4 speakers, timestamps spanning from 2.8 seconds to 2167 seconds, seamless.

6. Results and Bill

Tech stack overview:

Component Purpose
MOSS-Transcribe-Diarize 0.9B Transcription + Speaker Diarization
LM Studio (gemma-4-26b + hy-mt2-7b) Minutes Organization + Subtitle Translation
DeepSeek API Fallback (currently purely decorative)
Flask Small web interface with three tabs
ffmpeg The eternal god (audio format normalization)

Finally, after sending the first set of minutes generated by this system to the group chat, colleagues expressed:

"Well written!" "So detailed!" "We should use this often!"

As for whether they will actually read them—that's another closed-loop story.

7. Real Feelings: It's Fast, But Not That Fast

After all the hype above, here are some honest truths.

Speed-wise: For a 36-minute recording, MOSS chunked transcription took about 31 minutes—basically a 1:1 time ratio. Whisper Large V3 processes audio of the same length in about 10-15 minutes. MOSS is nearly twice as slow.

The reason isn't hard to understand: an end-to-end multi-task model (transcription + diarization done simultaneously) is inherently heavier than a pure transcription model, plus the extra overhead from chunking. The slowness was expected. Still, seeing the time after the first run, I paused for a moment—"Why so long?" Then I glanced at the progress bar, confirmed it wasn't frozen, and felt relieved enough to make a second cup of tea.

VRAM-wise: MOSS's VRAM usage is dynamic—short audio might only need 2-3GB, but a 36-minute long audio, expanded into tens of thousands of tokens, plus the attention matrix, heads straight for a VRAM explosion. This is why I had to implement the chunking scheme: without chunking, it simply couldn't run.

Even worse, MOSS and gemma cannot fit together on 16G VRAM—loading both models simultaneously instantly blows up the VRAM. So the actual workflow can only be serial: run MOSS transcription first, wait for it to completely finish and release VRAM, then load gemma for minutes organization. Throughout the entire process, MOSS and gemma never meet, like two security guards on different shifts, never seeing each other.

So the actual state of this solution is: MOSS transcription is slow, gemma loading is heavy, the two cannot run in parallel, everything is queuing. But the good news is it can run through; the chunking scheme withstood the VRAM pressure, preventing the whole process from crashing.

So the question arises: Why didn't I just use Whisper + a third-party diarization tool?

Actually, the Whisper ecosystem already has mature speaker diarization solutions, like the whisper-diarization project, which is a combo of Whisper + pyannote.audio. Theoretically, it can simultaneously achieve high-quality transcription and speaker labeling, and might even be faster.

I plan to try it out later. If the results are good, I might switch back. After all, in tech selection, there is no best, only what's most suitable for the current stage.

When I have results, I'll report back—maybe it'll be material for another article.

Final Words

The technical depth of this article is a bit higher than my previous ones. But the core idea remains unchanged: use limited hardware to do useful things. The 5060 Ti 16G might not be top-tier, but with reasonable allocation and clever scheduling, it can still run an end-to-end meeting minutes system.

Of course, this solution isn't perfect. Speed isn't fast enough, VRAM is tight, and MOSS and gemma cannot run in parallel, leading to longer total time. But for me, being able to run through, deliver, and save two hours of minutes organization time has already paid back the ticket price.

If the Whisper + diarization solution works out later, I'll come back with an update. Maybe the title will become "I Switched Back to Whisper Because MOSS Was Just Too Slow"—face-slapping is also part of the tech blogging journey.

If you're also fed up with the century-old problem of "who takes the meeting minutes," feel free to copy this homework.

If this article made you smile, or saved you an afternoon of organizing minutes, likes, comments, and reposts are all welcome.

Thank you all 🙏
May your recordings always be clear, your VRAM always sufficient, and your meetings never need a second round.

Comments

Top 1 of 2 from juejin.cn, machine-translated. The original thread is authoritative.

ivenshine

Enterprise meeting software nowadays all have corresponding AI meeting minutes.

雪隐_上班了

You're talking about large enterprises holding online meetings. For small businesses, just find an office with a screen and you can discuss requirements.