跪拜 Guibai
← Back to the summary

Why My Scraping Script Never Steals Your Cursor

I run several data-scraping scripts on my computer every day. They need to open a browser, log in, and click around on pages.

But they never appear on the screen. Not because they are hidden; there is simply no visible window at all. The only trace you can notice is an extra icon in the taskbar.

This sounds simple, but it's full of pitfalls. Here are the six I stepped on, in the order I hit them.

First, the counterexample: the default behavior of most scripts

When writing automation scripts, almost everyone's first version looks like this:

const browser = await puppeteer.launch({ headless: false });  // or spawn a Chrome

The reason for headless: false is legitimate: some sites' anti-scraping measures detect headless browsers, so using a real window is more reliable.

The cost is: a window pops to the foreground and steals your focus. You're typing in a document, and the next paragraph goes into the address bar.

So I changed my goal to one sentence: the script has to get the job done, but while you're using it, you shouldn't know it exists.

Three rules

There are only three rules, but each one comes with its own pitfall.

  1. Newly launched browsers start off-screen, then immediately minimize; no visible window appears at any point.
  2. Every subsequent run reuses that same browser, and none of the window state is touched, not a single byte.
  3. The only exception: when the login session expires and a QR code scan is needed, the window is brought to the center of the screen once, then put away after the scan.

Pitfall 1: --start-minimized no longer works

The most intuitive approach is to add --start-minimized to the launch arguments:

const args = [
  `--remote-debugging-port=${port}`,
  `--user-data-dir=${profile}`,
  '--start-minimized',        // ← modern Chrome ignores it
];

In practice: with this flag, the window still displays normally in the foreground. This parameter no longer has any effect on modern Chrome.

What actually minimizes is CDP's Browser.setWindowBounds:

export async function applyMode(b, targetId, mode) {
  const { windowId } = await b.send('Browser.getWindowForTarget', { targetId });

  // First pin the "restore position" inside the screen, otherwise when the user clicks the taskbar icon later, the window will move off-screen.
  await b.send('Browser.setWindowBounds', {
    windowId, bounds: { ...NORMAL, windowState: 'normal' },
  });
  // Then minimize immediately. The two steps are back-to-back, with no rendering await in between.
  await b.send('Browser.setWindowBounds', { windowId, bounds: { windowState: 'minimized' } });
}

A note on why the first step is necessary: if you only minimize, the window's "restore position" remains the off-screen coordinates from launch (-32000, -32000). When the user clicks the taskbar icon, the window appears off-screen, which looks like "I clicked and nothing happened", even more annoying than a popup.

Pitfall 2: Minimizing the window doesn't affect scraping (counter-intuitive)

At first I thought a minimized window would be frozen by Chrome and the page would not be able to fetch anything. So I came up with a bunch of solutions: off-screen, virtual desktops, switching to another workspace.

Test result: after the window is minimized, the page still renders normally.

The prerequisite is not to enable background throttling at launch. These three parameters are required:

'--disable-renderer-backgrounding',          // don't degrade rendering just because the window is invisible
'--disable-background-timer-throttling',     // don't reduce timer frequency to once per minute
'--disable-features=CalculateNativeWinOcclusion',  // don't treat an occluded window as invisible

So "not disturbing" and "getting the job done" are not in conflict at all. What conflicts is "not wanting to change the code."

Pitfall 3: When reusing the browser, don't do a single window operation

This is the easiest mistake to make and the hardest to track down, because it doesn't happen every time.

The typical flow looks like this:

// executed on every run
if (!await isAlive(port)) launch();
const page = await connect();
await page.bringToFront();      // ← the culprit
await makeWindowVisible();      // ← and this

The people who write these two lines (including me) have a good reason: what if the window is minimized and the page operation fails?

It won't. See pitfall 2.

But the actual effect of these two lines is: every time the script runs, it pops the window back up that the user manually minimized.

The correct approach is to make "which run is this" an explicit state: only when I launched the browser this time do I touch the window.

export async function ensure(cfg) {
  const ver = await portAlive(cfg.port, 1200);

  if (ver) {
    const st = readState(cfg.stateFile);
    // port is alive + the pid is still the process I recorded → that's "my browser"
    if (st && st.port === cfg.port && pidAlive(st.pid)) return { launched: false };
    throw new Error(`Port ${cfg.port} is already occupied by another program`);
  }

  launch(cfg);            // only launch when we get here
  return { launched: true };
}

The caller decides based on this:

const { launched } = await ensure(cfg);
if (launched) await applyMode(cdp, targetId, 'min');   // only set window when newly launched
// on reuse: do nothing

Pitfall 4: Port 9222 does not equal "your browser"

9222 is the default port for everyone. Your machine likely already has another automation tool (or a forgotten DevTools) connected to it.

If your script only checks whether the port is open, it might be pressing Enter in someone else's browser. The worst case I've seen: the script posted content under someone else's login session.

So port probing must add an identity check: write the pid from launch into a state file, and before the next use confirm "this pid is still alive, and it still occupies that port":

export function pidAlive(pid) {
  try { process.kill(pid, 0); return true; } catch { return false; }
}

Only reuse when both are true; otherwise, error out directly. Don't do "best effort". Better to let the script fail than to operate on someone else's browser.

Pitfall 5: Only show the window when logging in

QR code login can't be avoided, but you can minimize its impact:

// only this moment makes the window visible
await b.send('Page.navigate', { url: loginUrl }, sessionId);
await showWindow(b, targetId, sessionId, WINDOW.ON);

// poll login state... (polling code omitted)
await applyMode(b, targetId, MODE);   // put it away immediately after scanning

Another subtle issue: don't judge login state by cookie names.

I encountered this on two platforms: Storage.getCookies sometimes returns an empty array at certain moments (when the page context is not ready), even though document.cookie clearly has content. Conversely, the sessionid that truly represents the login state is often httpOnly, so document.cookie can't see it at all.

The most reliable criterion: send a real, read-only business request inside the page and look at the error code it returns. Whether the browser itself carries the login state is more reliable than any cookie name.

Pitfall 6: CDP flatten mode

This isn't related to "not disturbing", but it can block everything before the first step.

Once flatten mode is enabled, page-level commands must first attach to a specific page:

const { targetId } = await b.send('Target.createTarget', { url: 'about:blank' });
const { sessionId } = await b.send('Target.attachToTarget', { targetId, flatten: true });

// every page command after this needs sessionId
await b.send('Runtime.evaluate', { expression: 'location.href', returnByValue: true }, sessionId);

Without it, you get an error that's very easy to misinterpret:

'Runtime.evaluate' wasn't found

It's not saying "the method doesn't exist"; it's saying "you haven't connected to that page".

Summary

Looking back, this whole thing isn't high-tech; it's just a few API calls. The hard part is deciding to constrain yourself:

One last boundary note: by default, these kinds of scripts only do read operations: scraping data, looking at leaderboards, organizing competitor information.

Write operations (posting content, changing settings) all require an explicit switch plus manual confirmation before executing, and before publishing, the full content to be posted must be printed out and reviewed. No matter how smoothly a script runs, it shouldn't leave any ambiguity about "what exactly did I post".

Just because it can read doesn't mean it should write.


Comments

Top 1 of 2 from juejin.cn, machine-translated. The original thread is authoritative.

组合搭配函数

Sharing one browser between the script and a human is a genuinely smart move. You attach via CDP and each does its own thing, never stepping on the other's toes. The login state comes straight from the browser, so you skip the back-and-forth of logging in repeatedly.

stormsha

Thanks. But once you actually try it, 'sharing' turns out to be the biggest pitfall: two scripts crammed into the same browser fight over not just the login state, but also tabs and windows. Today I hit a whole string of them: one script's --stop killed the entire process tree, and another platform's scraper got ECONNREFUSED; a cleanup script also proactively closed all editor tabs, destroying someone's half-filled content. The current approach is 'one browser per platform': CDP port, --user-data-dir, and pid state file each get their own set, and you scan the login state once per platform. Sharing only pays off under one condition: you run exactly one thing at a time. Reusing the login state is indeed nice, that part is correct.