Screenshot 100 websites in one run, and get alerted when a competitor's page visually changes - DEV Community

DEV Community

Lindenbyte
Lindenbyte

Posted on

Screenshot 100 websites in one run, and get alerted when a competitor's page visually changes

Disclosure: I'm Lindenbyte, the small studio that built the Apify Actor used below, and this post was
drafted with AI assistance. For one page, your browser's devtools ("Capture full size screenshot") or Apify's
free one-URL screenshot Actor are fine. This Actor adds many URLs and devices per run, PDFs, and a memory of
the previous run so it can tell you what changed.

Screenshots are boring until you need fifty of them: a client report on desktop and mobile, a redesign
before/after, or proof of what a competitor's pricing page said last Tuesday. By hand, that is one tab, one
devtools command and one file rename per page and device.

Below are seven real runs (2026-11-13) of the
Bulk Website Screenshots Actor: a bulk capture, a
cookie-banner check, PDFs, and a two-run visual monitor, plus the Python that puts it on a schedule.

1. Capture a list of sites on desktop and mobile

{
  "urls": ["https://example.com", "https://stripe.com", "https://www.bbc.com",
           "https://crawlee.dev", "https://docs.apify.com"],
  "devices": ["desktop", "mobile"],
  "fullPage": true,
  "format": "jpeg",
  "quality": 80,
  "savePdf": true
}
Enter fullscreen mode Exit fullscreen mode

Result: 10 screenshots and 5 PDFs in 220 seconds, all HTTP 200. Each row has the URL, device, title, image
size and a screenshotUrl you can open without logging in.

The sizes are worth knowing before you scale up. example.com on desktop is a 63 KB JPEG; the BBC homepage scrolled
to 13,316 px and came out at 2.3 MB. Stripe's homepage (15,639 px) was cut at the default 15,000 px limit,
and the row says so in warnings. Mobile shots are rendered at 2x like a real phone, which means a hard ceiling of
8,192 CSS pixels: BBC (17,483 px), Stripe (20,880 px) and docs.apify.com (10,008 px) were all cut there on mobile.

Cost of this run: 10 x $0.003 + 5 x $0.0015 + $0.005 start (2 GB) = $0.0425.

2. Cookie banners: on vs off (best effort)

Banner hiding is on by default. To see what it does, I captured three news homepages above the fold twice, once
with "hideCookieBanners": false and once with the default true:

  • BBC: with hiding off, a "Terms of Use & Privacy / Accept and Continue" dialog covers the middle of the page. With hiding on, the same page is clean.
  • The Guardian: no consent banner at all from the Apify server we ran on (consent popups depend on the visitor's country), so nothing to hide. Its large year-end fundraiser box is page content, not a cookie banner, and stays.
  • Le Monde: the consent wall stayed even with hiding on. It is a custom one, not one of the big consent platforms the Actor knows.

The fix for the last one is one line: look up the wall's CSS class (right-click, Inspect) and hide it:

{
  "urls": ["https://www.lemonde.fr"],
  "fullPage": false,
  "hideSelectors": [".gdpr-lmd-wall"]
}
Enter fullscreen mode Exit fullscreen mode

The next run produced a clean Le Monde homepage. Banners are hidden with CSS, never accepted or rejected, so
treat it as best effort and keep a hideSelectors list for the sites you care about.

3. PDFs for the archive

savePdf adds one PDF per URL (not per device), A4 by default. From run 1: example.com is a 1-page, 48 KB PDF,
crawlee.dev 3 pages, docs.apify.com 7 pages (365 KB). Image-heavy pages get big fast: the BBC homepage PDF was
46 MB and Stripe's 16 MB.

For pages you want to read later (policies, docs, articles), try "pdfMedia": "print", which uses the site's
print stylesheet instead of the screen layout. docs.apify.com printed to the same 7 pages either way (345 KB vs
365 KB); Le Monde's print version was 8 pages and 9.1 MB. PDFs cost $0.0015 each and are not charged for error
pages.

4. Turn it into a visual change monitor

Monitoring mode keeps a small grayscale thumbnail of every URL and device under a stateKey and compares the
next run against it. changePercent is the share of pixels that differ; isChanged is true at or above
changeThreshold (default 1%). I watched two news homepages and two static pages, above the fold, with ads blocked:

{
  "urls": ["https://www.theguardian.com/international", "https://www.bbc.com",
           "https://example.com", "https://crawlee.dev"],
  "devices": ["desktop"],
  "fullPage": false,
  "blockAds": true,
  "compareWithPrevious": true,
  "changeThreshold": 1,
  "stateKey": "tutorial-08"
}
Enter fullscreen mode Exit fullscreen mode

The baseline run (11:58 UTC) returned four screenshots with changePercent: null: nothing to compare yet.
The same input 13 minutes later (12:11 UTC):

Page changePercent isChanged
theguardian.com/international 1.5 true
bbc.com 0.39 false
example.com 0 false
crawlee.dev 0 false

Opening the Guardian's screenshotUrl next to its previousScreenshotUrl shows what moved: two of the three
featured story cards under the masthead had been replaced. The BBC's 0.39% is small stuff (timestamps and
images shifting) and stays under the 1% threshold. The static pages are a clean 0: no false alarms.

One honest warning from the next run: a third capture of the Guardian, about 20 seconds later, showed
7.81% "change". No news moved; the fundraising box simply rendered a different layout variant. Pages with
A/B tests, carousels or live counters will trip any pixel diff. Put those elements in hideSelectors or raise
changeThreshold for that monitor.

5. Schedule it and grab only the changed pages

For a real monitor, switch on onlyChanged: unchanged pages are still captured and compared, but not saved
or output, and cost $0.001 instead of $0.003. Here is the whole loop with the official client
(pip install apify-client, version 3.x):

import os
from decimal import Decimal
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("lindenbyte/bulk-website-screenshots").call(
    run_input={
        "urls": ["https://www.theguardian.com/international", "https://www.bbc.com",
                 "https://example.com", "https://crawlee.dev"],
        "devices": ["desktop"],
        "fullPage": False,
        "format": "jpeg",
        "blockAds": True,
        "onlyChanged": True,          # output (and pay for) changed pages only
        "changeThreshold": 1,
        "stateKey": "tutorial-08",    # same name = same history
    },
    max_total_charge_usd=Decimal("0.50"),  # hard cost cap for this run
)

for row in client.dataset(run.default_dataset_id).iterate_items():
    print(f'{row["changePercent"]:>6}%  {row["url"]}')
    print(f'        now:    {row["screenshotUrl"]}')
    print(f'        before: {row["previousScreenshotUrl"]}')
Enter fullscreen mode Exit fullscreen mode

On my run it printed one row (the Guardian, 7.81%, with both image links) and skipped the other three. The
run cost $0.011: one screenshot, three unchanged checks and the start fee.

You don't need to host this script: save the input as a task in the Apify Console, add a schedule (daily or
hourly) and an integration (email, Slack or a webhook) that fires when the run finishes. A daily check of these four pages costs at most
$0.017 a day (all four changed), or about $0.50 a month.

Limits (honest ones)

  • Cookie-banner hiding is best effort (see Le Monde above). Logged-in pages are not supported in this version.
  • Sites with aggressive bot protection can answer 403 or a challenge page. Those rows are flagged and free; CAPTCHAs are not solved.
  • Long pages are cut (15,000 px by default, 8,192 CSS px on 2x mobile), and flagged.
  • Image-heavy full-page captures are slow: the BBC homepage took over a minute on its own. Budget roughly 15-25 seconds per heavy full-page shot at the default concurrency of 2.
  • Change detection compares pixels, so it tells you that and how much a page changed, not what the new text says. Open the two screenshot links to see the difference.

Not affiliated with any of the websites captured above.

Top comments (0)