Qwen 3.8 Max Burned a 5-Hour Token Quota on One Flutter Task That a Free Model Finished in 3 Minutes
Hello everyone, I'm Lao Liu
A couple of days ago, Lao Liu spent the price of a cup of coffee to buy the lowest tier of Qwen's token plan, ready to test whether Qwen 3.8 max could serve as the main workhorse in my Flutter project.
The task was simple: I had already developed three independent data processing and business logic modules, and now I just needed to integrate them into the main workflow so the main process could call them.
For this one job, Qwen 3.8 max ran for 20 minutes, directly burning through my 5-hour quota, and the work still wasn't finished. It stopped halfway.
I switched to opencode's free model, and the same task completed successfully in 3 minutes and 17 seconds.
Unbelievable, right?
This is absolutely not the performance you'd expect from a model aiming to compete with the top tier.
Today, Lao Liu will talk about what exactly happened during those 20 minutes, and the biggest pitfall we encounter when choosing a primary workhorse model.
To be fair, Qwen 3.8 max's benchmark scores are genuinely impressive
First, I need to clarify that the evaluation data Lao Liu is talking about isn't the kind from public benchmarks where scores can be gamed.
I've basically stopped looking at those, because I've been burned a few times: high scores, but terrible real-world working ability.
I now mainly look at benchmarks with non-public questions, like LiveBench.
Qwen 3.8 max scores very high in these kinds of evaluations, which shows its real capability is noteworthy.
As you can see, Qwen 3.8 max's overall capability is second only to top-tier models like Claude Fable 5 and GPT-5.6.
Looking solely at coding ability, it's also very good:
Evaluations with closed-source questions have relatively high reference value and can, to a certain extent, reflect a model's generalization ability in a given subject.
But note that this is only reference value, because there's still a significant gap between benchmarks and real-world projects.
Why can't benchmarks save your project?
Scenario mismatch
Lao Liu mainly writes Flutter code, but current mainstream coding benchmarks basically don't use Flutter.
Using a score from a Python and Web exam to predict its Flutter proficiency is like using a driver's license written test score to judge whether someone can drive on a mountain road.
Usage mismatch
Lao Liu's daily development workflow is like this: first, use a top-tier model to break down a requirement into multiple feature points, then use a mid-range primary model to develop the code for each feature point one by one.
This usage actually falls somewhere between LiveBench's coding and agentic coding categories.
In other words, for developing these feature points, no single dimension from the benchmark is suitable for choosing a model. Even using an average score, the deviation is large.
So benchmark scores are mainly a reference; ultimately, you have to run it in a real project.
Lao Liu did exactly that this time—a real, live-fire test.
Qwen 3.8 max was simply too diligent
If I had to sum up Qwen 3.8 max's performance in my project in one word, it would be verbose.
Looking at its thought process reveals the problem: the reasoning chain is extremely long. If the previous round's reasoning is added to the input each time, the tokens are consumed very quickly.
This is also confirmed by LiveBench's cost-effectiveness ranking:
Looking solely at cost-effectiveness for simple coding, Qwen 3.8 max's ranking is not high.
So Lao Liu's conclusion is: Qwen 3.8 max's capability is indeed good, but the cost is that it needs to consume a huge number of tokens to complete the same task.
It's like hiring a very capable employee, but they burn through a day's worth of someone else's salary for an hour's work.
Judging from the benchmarks, its cost-effectiveness in logical reasoning, data processing, etc., is significantly higher than in programming. Using it as a primary coding workhorse might be a mismatch of specialties.
Interestingly, the typo corrections, formatting optimization, adding fixed header/footer content, and publishing to GitHub for this very article were all completed by Lao Liu using the remaining quota from the token plan, via Qwen 3.8 max. The total consumption was far less than the development task.
This also indirectly proves Lao Liu's point: Qwen 3.8 max seems more suitable for non-coding tasks.
A lesson learned for the price of a cup of coffee
To be honest, Lao Liu himself was a bit embarrassed by this test.
I bought this lowest-tier token plan originally intending to horizontally test which vendor's subscription is better, and I'll report back to everyone on the usage experience of other vendors' subscriptions later.
The result with Qwen was that the first long conversation hadn't even finished running, and the quota was already dead.
I have to complain here: even though the lowest tier only costs as much as a cup of coffee, if a 5-hour quota can't even complete a single long conversation, then it has absolutely no practical meaning.
Especially when many vendors like Agnes, SenseTime, OpenRouter, opencode, etc., offer generous free quotas, this kind of low-tier subscription completely lacks practical value in a development scenario.
Final words
This failure has made Lao Liu even more convinced of one thing:
Benchmarks are for outsiders to see; the bill is for yourself to see.
When choosing a primary workhorse model, besides looking at the capability ranking on leaderboards, you also need to see how much it costs to complete each task.
Figure this question out first, then go look at the benchmarks. The order must not be reversed.
Finally, Lao Liu wants to ask everyone: Have you ever been fooled by benchmarks? What do you value most now when choosing your primary model? Feel free to chat in the comments.
🤝 If any readers here are interested in client-side or Flutter development, feel free to contact Lao Liu. Let's learn from each other.
🎁 DM for a free copy of Lao Liu's "Flutter Development Handbook," covering 90% of application development scenarios. It can serve as a knowledge map for learning Flutter.
💬 : laoliu_dev
📂 Lao Liu has also organized his historical articles in a GitHub repository for easy reference.