跪拜 Guibai
← Back to the summary

Three Chinese Multimodal Models Go Head-to-Head on Real Production Tasks

Now, the demands on foundational model capabilities are getting higher and higher. People are no longer satisfied with text input alone, but rather mixed input.

For example, an interface screenshot, a business chart, a scanned contract, an invoice, etc. The model must not only be able to understand the content within the file, but also be able to turn that content into executable steps, extract structured information, or provide business conclusions.

In other words, the requirements for multimodal understanding and execution capabilities have increased. Currently, several domestic models that support multimodality include Step 3.7 Flash, Qwen3.6-flash, MiniMax M3, and others.

So, how do these multimodal models actually perform in a production environment? Can they really be used in production?

In the past, everyone liked to look at benchmark leaderboards. To be honest, I suspect many people don't care that much about the scores. Many models have brilliant public scores, but when it comes to specific tasks, they fall flat.

Here, we're not playing games either. We'll directly use real-world tasks for a side-by-side comparison of multimodal understanding and execution capabilities.

Evaluation Description

To ensure fairness in testing, I'll lay out the evaluation methods and criteria first, without favoring any model. If it works, it works; if it doesn't, it doesn't. We'll only speak with facts and data.

Here, we ensure that under the same task scenario, we use the same prompts, the same configuration parameters, and the same tools or methods to test the execution of different models. The only variable is the model. And here, the execution results, time, and token consumption for each task are all presented together.

Ultimately, we're looking at three main points:

  1. Quality: Can a single conversation yield a usable result, or does it require repeated follow-up questions?

  2. Speed: Is the end-to-end return fast enough to be suitable for high-frequency calls by an Agent?

  3. Cost: The model's unit price and token consumption, and whether subsequent manual investment is needed.

Because evaluating whether a model can be used in a production environment comes down to these three key dimensions.

Below, I've selected two cases, both of which are scenarios where the company currently uses multimodal models at high frequency. One is used within an Agent, and the other is used via API in a business scenario.

Scenario 1: Reconstructing Business Logic from a Flowchart

Before writing code, the front-end and back-end teams design the entire technical solution together. This step inevitably involves system architecture diagrams or business flowcharts. Coding only begins after the plan is finalized. Now, with the support of multimodality, we can completely hand the finalized flowchart to the AI to extract the solution logic and formulate an implementation plan.

For example, we need to implement login authentication for a WeChat Mini Program according to the logic of the following flowchart, which has a total of 10 steps (counting the arrows):

First, put the screenshot into the project file, then open Claude Code and enter the following prompt:

@WeChat_Mini_Program_Login_Solution.png Get the process logic from the image

First, the output from Step 3.7 Flash:

Step3.7-flash provided the participants, the complete process, and the core design ideas. It identified the complete process as 10 steps, with the total number of steps and the logic of each step perfectly matching the original flowchart. The output quality here is top-notch. It can not only recognize the text in the image, but also understand the process logic within the image, expressing the flowchart's logic completely and accurately.

Below is the output from MiniMax M3:

MinMax M3's output was also a total of 10 steps, with the logic for each step being correct.

Next, let's look at the output from Qwen3.6-flash:

Qwen3.6-flash's output was a total of 9 steps, one step fewer than the reference standard. It merged steps 3 and 4, but the overall logic was correct.

At the bottom of each task above, there is the corresponding model name, execution time, and token consumption. Here, I've organized the execution data for each model into a table:

Dimension Step 3.7 Flash MinMax M3 Qwen3.6-flash
Time (API Time) 15s 20s 19s
Token Consumption (Input, Output, cache Read, cache write) 728, 1.1k, 54.4k, 0 27.9k, 1.2k, 228, 0 251, 1.9k, 0, 28.6k
Token Price Conversion ¥0.0246 ¥0.0688 ¥0.0483

In this scenario, if we only look at output quality, there is actually no significant difference between the three models. But according to the criteria we set earlier, Step 3.7 Flash's performance is more advantageous: it's faster, cheaper, and has stable generation quality.

This scenario is well worth trying. There are a lot of business flowcharts in development. Previously, we had to spend a lot of time describing them verbally to the AI, and it was prone to errors. Now, with the help of multimodal models, we can save a lot of time and cost while ensuring quality.

Now, let's look at another case of using a multimodal model in a business system.

Scenario 2: Using Multimodality to Assist an Invoice Entry System

In our business, there is a step for entering invoices into the system. The process is to first take a photo and upload it, then manually enter the information from the invoice into the system form in sequence. This operation is time-consuming and labor-intensive. Previously, we wanted to solve this through OCR recognition, but the recognition error rate was high, and manual confirmation was still needed. The problem with OCR recognition is that it mechanically identifies information.

Currently, we are optimizing this with the help of multimodal models. Their advantage is that they can not only understand information but also reason about it.

Below is an electronic invoice. The task is to have the model recognize the information in the invoice and output the key field information in a structured way, so it can be automatically entered into the system, reducing manual entry costs.

Because this is called via API in a business system, a test script was written, where the image address and prompt are consistent.

The prompt is set as follows:

Please extract structured information from this invoice image and return it according to the following JSON structure:
{
    Invoice Type: string,
    Invoice Number: string,
    Invoice Date: string,
    Invoice Amount: string,
    Tax Rate: string,
    Tax Amount: string,
    Item Name: string
    Buyer's Taxpayer Identification Number: string,
    Buyer's Bank: string,
    Seller's Name: string,
    Seller's Taxpayer Identification Number: string,
    Seller's Bank: string,
}

Below, we'll still follow the previous order and look at each model's performance in turn.

First, let's switch the model to Step 3.7 Flash and run the script:

From the results, Step 3.7 Flash's extraction was completely correct, taking 5.6s and consuming a total of 1409 tokens.

Next, let's switch the model to MinMax M3 and run the script:

From the results, MinMax M3's extraction was also completely correct, taking 6.1s and consuming a total of 2216 tokens.

Now, let's switch the model to Qwen-3.6-flash and run the script:

From the results, Qwen's performance was also very stable, similar to the previous two models, with no extraction errors. The total time was 7.38s, and the total consumption was 2008 tokens.

Here is a summary of each model's execution:

Dimension Step 3.7 Flash MinMax M3 Qwen3.6-flash
Time (API Time) 5.6s 6.1s 7.38s
Token Consumption (Input, Output, cache Red, cache write) 802, 607, 0, 0 1686, 530, 1672, 0 1165, 843, 0, 0
Token Price Conversion ¥0.0060 ¥0.0086 ¥0.0075

In this scenario, there was still no difference in the generation quality among the three. All could accurately extract the information from the invoice according to the required JSON structure. However, in terms of response speed and token consumption, Step 3.7 Flash still had a slight advantage.

This recognition cost can be said to be very low. The cost of structured information extraction for one invoice is less than 1 cent, and on this sample invoice, all three could correctly extract the fields. Therefore, applying multimodality to the scenario of structured invoice information extraction is very worth trying.

Summary

The two cases above, one used in an Agent and the other in a business interface, actually verify the same thing: after receiving complex visual input, can the model transform it into a directly usable result according to our requirements?

Here, I've organized the actual test results from the two scenarios into the table below:

Dimension Step 3.7 Flash MinMax M3 Qwen3.6-flash
Speed Fast Medium Medium
Token Consumption Low High Medium
Token Price in RMB Cheap Expensive Medium
Stability Excellent Excellent Excellent

Overall, in both scenarios, the quality stability of all three models was very good, with no extraction errors. However, if placed in a production environment, quality is only the first hurdle. After that, you also need to look at response speed, call cost, and suitability for high-frequency access.

Considering these dimensions, I would personally be more inclined to prioritize testing Step 3.7 Flash in an Agent or business API. It maintained good output quality in both scenarios, while being faster and having lower token consumption, making it more aligned with the requirements for 'production-ready'.

Of course, this doesn't mean one model can cover all scenarios. Before truly going live, you still need to run a round of tests with your own business samples.

Alright, the above evaluation only represents my personal actual testing. Everyone can also use similar tasks to experience the overall multimodal understanding and execution performance of Step 3.7 Flash.