Rebuilding JSON from 4KB logcat shards when encryption breaks packet capture
Encrypted interceptors make packet capture useless, so I had to reassemble JSON from 4KB logcat fragments
Android debugging's real pipeline: after the network-layer interceptor does encryption and decryption, packet capture only shows ciphertext. The one plaintext line in logcat becomes the only place you can see the final shape of the API. But it has a 4KB limit and can lose the middle segment. Generic online JSON tools, on the other hand, assume you have complete data in hand.
When I match fields with the backend, I open Charles and get ciphertext.
Back in logcat, the plaintext is there, but cut into five segments, each prefixed with a timestamp, pid, and package name.
This is the Monday-morning routine of Android integration. It used to take five minutes per round; now it's ten seconds. Try it
1. Why packet capture isn't enough
The network layer has an Interceptor that encrypts request bodies and decrypts response bodies. This is necessary for security and risk control, but the cost is that packet capture tools become useless.
Charles with the certificate can see HTTPS plaintext, but the body is ciphertext, a base64 string. Replaying the packet also fails, because signatures don't match and the server returns an auth error.
So in the whole chain, the only place you can see the final form of business data is one line of logs: after decryption, before serialization.
logcat here is not a fallback; it's the only window.
This also explains why many Android developers, even with packet capture tools, instinctively open the log when a field is off. It's not habit; it's that the data is only readable there.
2. This window is very narrow
Three constraints stack together:
A single message has a length limit. About 4KB; the system cuts it off when exceeded. The cut position ignores syntax, so it can fall right in the middle of a key.
The buffer is a ring buffer. When full, old data is overwritten. The overwrite happens in the middle, not at the end.
Every segment carries a prefix. Timestamp, pid-tid, package name, level, tag; each cut repeats all of them.
The result: you want a JSON document, but you get several fragments, each with a line of metadata in front.
3. So integration becomes this
Walk through the manual flow and you see where the time goes.
Step one: copy. Precision matters; selecting one line too many or too few breaks it. A missing line might contain the closing }.
Step two: paste into an online tool. It reports a format error.
Step three: go back and delete the timestamp and package name from each line. Here you have to be careful, because the package name is immediately followed by {; delete one extra character and the JSON breaks.
Step four: paste again. Still wrong, because a middle segment is missing, and deleting prefixes doesn't fix that.
Step five: give up. Return to Android Studio, eyeball the fields, and hand-type the data class one by one.
An interface has a dozen fields, nested three or four levels deep. Halfway in, you notice one missing and go back. Rename a field and type it again.
During integration this happens a dozen times a day. Each round is three to five minutes, adding up to nearly an hour a day.
Put the time under a microscope:
| Step | Manual | With the tool |
|---|---|---|
| Copy + delete prefix | 1 to 2 minutes, careful not to delete too much | Paste as-is |
| Format | Half a minute, fails if a chunk is lost | Instant |
| Type data class | 2 to 3 minutes, a dozen fields | Copy directly |
| Per round | 3 to 5 minutes | About 10 seconds |
| 15 times a day | Over 50 minutes | Under 3 minutes |
4. The boundary of generic tools is the opportunity for vertical tools
Tools like this aren't wrong; the problem is their assumption: the input is a complete, valid JSON document.
That assumption holds for frontend and backend; everyone there has a directly usable chunk of data. But for Android it doesn't hold: what you have is fragments fished out of a log buffer, with timestamps stuck on the front and possibly a missing middle.
Sites like json.cn aren't the only ones; the first pages of search results are all similar: format, compress, validate, convert to XML, convert to YAML, JSONPath. They serve the same general need, so none of them goes deep for any one scenario. In other words, switching to another one is the same.
The three things I needed, none of them had:
Recognise a prefix like
1234-5678/com.pkg D/OkHttp:and know1234-5678is one unitSee that
pe":"GLOBAL"lost something before it, and rejoin the correct nestingGenerate a Kotlin data class directly, with annotations, nullability, and naming conversion
It's not that they're poorly made; it's that they don't serve me. That's category division, not a capability problem.
Ads follow the same logic. Generic tools live on traffic, and traffic needs ads to monetize; what I need is something that opens and works, doesn't require switching to a browser, and keeps data on the machine. The two goals are not aligned, so every time I switch over, it's a compromise.
What's really missing isn't a "better JSON formatter"; it's "the one for Android integration".
5. So I built one
Written with AI assistance, a single HTML file, offline, double-click to open. It ships with sample data, so you know what it can do without preparing a log first.
No Node setup, no environment configuration, no account registration. The UI is two boxes: paste logs on the left, results on the right. Nothing to configure. You can hand it to a test engineer; they have logs from the production package and can reconstruct fields themselves, without pinging you every time.
The core is four things:
| Part | What it does |
|---|---|
| Strip prefix | Matches seven logcat copy formats, in order of decreasing information |
| Reassemble fragments | Scans bracket and quote balance; concatenates only when same tag and previous line is unclosed |
| Permissive parse | Handles trailing commas, single quotes, unquoted keys, and Data(a=1, b=2)-style toString output |
| Generate classes | Splits Int/Long by int32 boundary; outputs Gson or kotlinx annotations |
One decision deserves a separate note: I did not use JSON.parse.
JSON.parse('{"orderId":9007199254740993123}')
// {"orderId":9007199254740993000}
JavaScript's Number is a double-precision float; its safe integer limit is 2^53-1. Nineteen-digit order numbers and user IDs get silently corrupted, and you'd never know. You'd just think the backend's ID is a little weird.
So the parser is my own recursive descent, keeping numbers as their original strings and deciding Int vs Long from the source text.
6. What AI got right the first time
The four things in the table above mostly worked on the first version.
Regexes, recursive descent parsers, template generation, Kotlin type mapping; these are all AI's comfort zone because the inputs and outputs are clearly defined, with no ambiguity. The whole tool is a single 1900-line file, zero dependencies, and it ran as soon as it was written.
The completion looked good at that point. Then I fed it a real log.
7. Three defeats by real logs
First: concatenation swallowed newlines
The early version simply joined fragments directly:
prev.content += line;
JSON doesn't care; whitespace doesn't affect parsing. But multi-line protobuf TextFormat gets compressed into one line, so the later "is this a multi-line structure" check fails and the entire segment takes the wrong branch.
The fix was to look at the balance scan state: if it broke inside a string, join directly; if it broke between tokens, add a newline.
Second: the permissive parser ate the next field's first letter
parseObject had an unconditional i++ at the end, intended to skip odd characters for error tolerance. When it encountered input without commas (protobuf style), it ate the first character of the next field name.
nickname became ickname.
This bug never triggers in JSON, which has commas. It only surfaces in another format.
Third: the log broke in the middle
This was the hardest.
I had assumed truncation only happened at the tail, and handled it simply: if the closing delimiter never arrives, return the parsed part and take what you can get.
But the real log looked like this:
..."accountDisabled":false,
17:45:15.895 D pe":"GLOBAL","rankType":"honor",...
What is pe? It's the tail of rankBadges. The middle chunk "rankBadges":[{"topN":1,... is not in any line.
When the ring buffer overflows, what's lost is an entire middle segment, not the end. Where it's lost depends on the moment you copy.
In an encrypted chain there's no fallback: packet capture sees ciphertext, and server logs are inaccessible. The dropped part is simply gone.
Joined together it becomes "accountDisabled":false,pe":"GLOBAL", and pe is treated as a new field name, shifting every subsequent key. The generated class then contains things like val 17: Int?, which cannot compile.
8. Two pitfalls in the fix
Detection: what counts as "something lost"
The left end stops at a token boundary, and the right end is not a legal token start; only then can it be a lost segment. A normal tail truncation stops at a string or number in the middle, which is different.
function isLostSegment(prevText, nextText) {
var left = prevText.replace(/\s+$/, '');
var right = nextText.replace(/^\s+/, '');
if (!left || !right) return false;
var lastCh = left.charAt(left.length - 1);
// left end must be on a token boundary, otherwise it's a normal truncation
if (lastCh !== ',' && lastCh !== '{' && lastCh !== '[' && lastCh !== ':') return false;
var firstCh = right.charAt(0);
// legal JSON token starts
if (firstCh === '"' || firstCh === '{' || firstCh === '[' ||
firstCh === '}' || firstCh === ']') return false;
if (firstCh === '-' || (firstCh >= '0' && firstCh <= '9')) return false;
if (/[A-Za-z_]/.test(firstCh)) {
var m = /^[A-Za-z_]\w*/.exec(right);
// xxx" → a truncated key tail, meaning middle content was lost; otherwise it could be true/false/null
if (m && right.charAt(m[0].length) === '"') return true;
return false;
}
return true;
}
The key is the last xxx": a letter directly followed by a quote. That shape never appears in valid JSON; it's evidence of a truncated key.
Realignment: why bracket-depth scanning fails
Once a lost segment is detected, you need to rejoin the following content at the right nesting level. The first approach was bracket-depth scanning: count { [ and } ] from left to right, and the first position where depth goes negative is "back to the outer level".
Sounded correct; in practice it was all wrong.
The reason is plain. The opening " of the surviving fragment was also lost. Originally "rankBadges" had matching quotes, now only a single closing quote remains. The entire text's quote parity flips, so every string start/end judgment is off by one. Internal {, }, and , are mistaken for structural tokens, and the depth calculation collapses.
There is no generic solution for this pit, because the information itself is gone. You have to find another approach that doesn't rely on quote pairing.
The missing segment must have swallowed the array's [, so the first ]," or ]} in the surviving fragment is the array's real end position. Skip the fragment by joining there:
function findRealign(text, start) {
var rest = text.slice(start);
var m = /][ \t\r\n]*,[ \t\r\n]*"/.exec(rest);
if (m) return start + m.index + 1;
m = /][ \t\r\n]*}/.exec(rest);
if (m) return start + m.index + 1;
// if the lost segment had no array, fall back to the object end
m = /}[ \t\r\n]*,[ \t\r\n]*"/.exec(rest);
if (m) return start + m.index + 1;
return text.length;
}
The effect: activity went from 74 fields back to 111, with lang, groupInfo, and subs all restored, losing only rankBadges — which is unrecoverable because it was in the dropped chunk.
9. The batch of Android-specific pitfalls
This part has nothing to do with AI; it's about the platform. People who build generic JSON tools never see it; only people who read logcat daily do.
Prefix formats are more numerous than you'd think. At least seven, depending on the Android Studio version, whether threadtime is enabled, and whether the text was copied from a table view. The easiest to get wrong is 1234-5678/com.pkg: the / is not a space, and the pid and package name are one unit; a regex that splits on spaces fails to match.
Long-text log continuation lines have no tag. When an in-house long-log framework prints a continuation, the shape is 17:45:15.895 D content, only time and level. If you can't identify it, the whole line gets swallowed as content along with the timestamp, and 17 and 15.895 become field names.
Class names collide. A field named list generates data class List(...), shadowing Kotlin's List and breaking compilation. A field named activity generates data class Activity, colliding with android.app.Activity. These errors are hard to spot because the class name is itself perfectly legal; it just hijacks a standard type name.
Now the approach is a blacklist: List, Map, String, Activity, View, Fragment, Context, Intent all get the Item suffix.
Nullability is a trade-off. The tool offers three levels: all-nullable, infer from samples, all-non-null; the default is all-nullable.
Inferring from samples sounds clever but is unreliable. You only have one sample; the field appears this time, but there's no guarantee the next response returns it, a server-side feature flag can make it vanish. So "non-null in the sample" doesn't imply "non-null in reality"; guessing wrong causes a production crash.
All-nullable costs ?: everywhere in business code. All-non-null costs an exception when a field is missing. The default is the former, because a parsing tool shouldn't make business decisions for you.
Protobuf has three forms. JSON mapping (JsonFormat.printer()) is naturally supported, but note that int64 in proto JSON is a string, which will be inferred as String. TextFormat (proto.toString()) also parses; the key is that repeated fields are expressed by repeated keys, not arrays, so merging is required. Binary hex can be decoded without a schema, but field names can only be field_N.
10. Where AI is blind
These three rounds of rework made the boundary fairly clear.
AI's blind spot is not "writing incorrect code"; it's not knowing the situation exists.
It writes regexes, parsers, and type mappings confidently, because these are deterministic tasks with clear inputs and outputs. But "the logcat buffer loses the middle segment" is not in its prior, nor in any requirements document. It only surfaced after I pasted a real log.
Similarly, the nickname → ickname bug is logically correct; the premise (that commas separate fields) just doesn't hold.
So I changed the workflow to this:
Real data before constructed data. The first version's samples were all valid JSON and exposed nothing. The moment a real log came in, three bugs surfaced at once. Test cases should grow out of real data, not be invented first.
Make the tool speak human. The status bar shows "log lines 4 / entries 1 / auto-joined 10 times / log missing 1 place skipped". This line saved me several times, because "joined 10 times" and "missing 1 place" are anomaly signals that surface trouble earlier than whether the result is right. If the tool handled it silently, I'd never know it was guessing.
Run tests outside the DOM. All parsers are pure functions; an export object is attached at the end of the IIFE. The test script mocks a minimal document and then evals the script inside the page. That way you test the exact production code, not a copy. Assertions grew from 42 to 86, and every parser change runs them first.
11. What AI truly changed: "is it worth writing"
The previous sections are about what AI can't do. But it did change one thing, just not at the code level.
Before, writing your own tool was uneconomical. To save fifty minutes a day, you'd spend two days writing code, nearly a month to break even. When requirements shifted mid-way you'd have to change it, then test again. Most people would rather tolerate the pain; they tolerate it for a year.
AI pushed the cost of the first version down to a few tens of minutes. It breaks even the same day, which makes "write a tool for myself" viable again.
This change is especially visible in Android development, because the small daily utilities are so numerous: logcat filtering, dp-to-px conversion, color conversion, JSON comparison, event parameter assembly, crash stack recovery. None of them deserves a project slot, but each one steals time.
These tools share a common trait: they are very vertical. Generic tools won't do them because the user base is too small to pay back the cost; but for you, they're used every day.
Now each can be built in half an hour, thrown into a folder, and opened with a double-click. No publication, no review process, no server.
The decision rule is simple: do I do this more than five times a week? If yes, it's worth half an hour; if no, keep tolerating it.
That's also why AI is a particularly good fit for these tools. They have no complex architecture, no concurrency, no scalability requirements; they just need to handle the boundary conditions cleanly in a narrow scenario. AI excels at writing this kind of deterministic code, and the rest is left to real data.
12. Why single-file, zero-dependency
I didn't package this as an npm package or add a build pipeline. One HTML file, double-click to open.
The reason is practical: the lifecycle of such a tool depends on whether it can still be opened. Anything with a build chain will likely fail npm install two years later; Node versions, dependency conflicts, lockfiles, any one of them can scare you off. A single file doesn't have that problem; as long as a browser exists in ten years, it will run.
Offline serves the same consideration. Data stays on the machine, and you don't worry about what third-party scripts on a page might be doing.
For me, the value of open source is not "free to use"; it's that these small tools survive on maintenance. If I stop using it and stop maintaining it, someone else can fork it and keep improving it, so it doesn't die.
If you only want the parsing logic: parseLog, parseLoose, typeOf, and genKotlin have zero DOM dependencies. Copy them out and they run; port them to a CLI (adb logcat | logcat2kt) or an Android Studio plugin.
13. Its boundaries
Without a .proto, there are no real field names. Binary decoding can only give field_1, field_2; that's an information-theoretic ceiling, not an implementation issue.
Mid-chunk loss can only be partially recovered. Fields that fall into the missing segment are never recoverable; the status bar reports how many places were lost, but it won't fake completeness.
Gzipped or encrypted bodies are out of scope. But encrypted data is already decrypted at the interceptor layer, so whatever makes it to the log is plaintext; this constraint rarely bites.
Mixed-type arrays degrade to List<Any>. Something like [1, "a", null] can't be inferred to a unified type and shouldn't be guessed.
Enums can't be inferred either. Whether code: 0 is an enum or a status code, the tool doesn't know.
Finally
If you touch an API twice a week, hand-rolling it is faster than configuring a tool, so use whatever works.
I wrote it because during integration I touch it a dozen times a day, and every time I have to switch between browser and IDE. The time saved is real: fifty minutes a day, a full day per iteration.
The chain is short: after the interceptor decrypts, data has only one exit, logcat. That exit is throttled by 4KB, gnawed through the middle by the ring buffer, and requires manual prefix deletion. The tool fixes exactly that section.
The next time a log bites me, I'll still be fixing the parser.