Research

Screenshots need their own PII detector

Snowmelt Labs research · March 2026

A computer-use agent filling in a checkout form leaves a trail of screenshots, each showing whatever it has typed so far, which might be half a street address or the first digits of a card, and before anyone shares them those values have to be redacted. The obvious approach is to extract the text with OCR and then classify what should be removed. We started there, but were also curious whether we could learn to locate the sensitive regions directly from screenshots, including fields that had yet to be filled.

We built WebPII, a dataset of about 45,000 synthetic e-commerce screenshots with every personal field boxed, and used it to train a small detector, WebRedact. On fully filled checkout and account pages from a retailer held out of training, the best OCR-based pipeline, LayoutLMv3 feeding GPT-4o-mini, reached 0.357 mAP@50 and took 2.9 seconds a screenshot. WebRedact reached 0.663 on the same pages in 20 milliseconds on a laptop CPU. It can also box a form field before anything is typed into it, which a text pipeline can't do. Averaging over empty and half-filled forms as well as full ones, it reaches 0.753, making it the approach we would use when collecting or releasing agent screenshots.

We have released the dataset¹1. WebPII: Benchmarking Visual PII Detection for Computer-Use Agents, arXiv:2603.17357 (2026). Figures 2 and 3 here are from the paper, CC BY 4.0. on Hugging Face along with the trained detectors (project page).

What does text extraction miss?

Take a street called Kelsey Villages and a brand called Kelsey. Once the page is reduced to strings, the two look much the same. On sparse pages, OCR sometimes misreads footer text or a company name, and the classifier, told to flag anything that looks like data, flags the misread string. Dense product listings fill the classifier's context instead, and it loses track of fields. Gift messages also get missed, even though they contain a note together with the sender and recipient.

What counts as personal?

We had to decide what personal data means on a store page. Beyond names and addresses, or card numbers, there are order numbers, tracking numbers, delivery dates and purchase histories. Most PII definitions leave these out, although they can still identify people. Four card transactions with merchant and date were enough to re-identify 90% of people in one study of card records.²2. Y.-A. de Montjoye, L. Radaelli, V. K. Singh and A. Pentland, "Unique in the shopping mall: On the reidentifiability of credit card metadata," Science 347 (2015). This is why we include fields that no existing benchmark labels, such as gift messages and B2B purchase order numbers (which tie a person to an employer). Delivery security codes belong here as well, since they let someone collect the package.

Learning from partially filled forms

If we only redact a field once it is complete, the screenshots taken while it was being typed still expose the partial value. So we render every form at each step of being filled. For a form with N fields, stage k has the first k−1 fields filled, field k half typed and the rest empty. This makes 28,653 of the images partial forms, against 10,200 full and 6,012 empty, so we can train on the states an agent encounters along the way.

We might expect that adding empty forms would help the model recognize fields before they contain anything. However, training on full and empty forms actually scored lower than training on full forms alone. Empty forms helped when we included partial ones as well, with that combination scoring higher than either. The number of intermediate stages also mattered, and going from 1 to 5 partial stages per layout raised the score from 0.758 to 0.802 (0.774 to 0.835 on partial-fill test images).

More layouts or more pages?

Layouts take work to make, so we wanted to know whether the next improvement should come from more layouts or more renders of the ones we had. Each layout is rendered about 120 times with different fake data and fill states, so increasing the page count does not necessarily give us more designs. We retrained on parts of the training set to separate these effects. Every score here is mAP@50 on the held-out retailer averaged over fill states, using fresh runs under one recipe³3. This is why the full-set YOLOv8n scores 0.739 here while the released detector scores 0.753.. The larger detector is YOLOv8s at 640 pixels, which is a different model from WebRedact-Large.

A random 10% of pages spread across 322 layouts scored level with the full training set, and 0.12 to 0.14 above runs on about as many pages from 33 layouts⁴4. 0.736 for YOLOv8n and 0.785 for YOLOv8s.. The 10% runs did train longer, because several 33-layout runs stopped early on a validation set of unseen layouts. Even allowing for that, the one 33-layout run that trained the full 100 epochs scored just 0.656 against 0.785. More renders of the same designs added little, while more designs added a lot, which suggests we should spend the next round of data collection on new page designs.

Every run drew from the same training companies, so we haven't tested whether new sites matter beyond that. Although the curve hadn't flattened at 331 layouts, the last step is one run per size, and for YOLOv8n the improvement is smaller than the spread between two draws at 33 layouts. The YOLOv8s full-set run was also cut at about half its schedule by a time limit. All of these scores come from one held-out retailer, 56 layouts, which limits how much we can infer about performance elsewhere.

How the data is made

To make this dataset, we needed realistic pages without putting real people's screenshots into the training data. We collected reference screenshots from the sites of 10 companies, nine of them stores, covering 19 page types (carts and checkout through order tracking, gifting and store pickup) and 408 layouts. A coding model rebuilt each layout as a React component in which every personal or product value is a variable carrying a data attribute. We can then render each layout with 25 data configurations and read the boxes straight out of the DOM, giving us pixel-accurate labels without hand annotation, or 993,461 boxes in all. The values come from a fake-data generator and an open product catalogue. Order totals are computed at render time so that the arithmetic on the page is right.

Rebuilding the layouts wasn't fully automatic. For 160 of the 408 layouts (39%), a person needed to send back a correction, with 312 corrections in total. These mostly addressed overflowing viewports or missing markup, as well as broken initialization. Collecting the source screenshots was the slow part, since we needed real signed-in sessions on each site.

Text in agent data has a similar problem. An agent-trace dataset we were about to ship held credentials our prefix-based scrubber had no rule for, and across three trace corpora 545 of 570 secrets had no known prefix to match (gitleaks, an open-source scanner with a full rule set, caught them before release).

Limits

The detector learns two classes, text and image, so it locates sensitive regions without telling us which kind of personal information each contains. Product images are easy across every split (0.855 to 0.875 AP@50), but text drops to 0.471 on the held-out retailer. A new page type is also harder than a new site, 0.728 against 0.753.

Everything here is English-language e-commerce, and extending it to other languages or kinds of site means generating more pages. Training uses static images, while real sessions scroll and change state. Finally, our taxonomy treats every product and order field as identifying, so a deployed model would over-redact without more context.