Where Clusterflick's Data Comes From

Where we sit

Every place, by what its robots.txt and its terms say about how we fetch from it. A site can object in either, so a clash is anything in the top row or the right-hand column.

How we ask

How we fetch from each place, from a plain request to a browser built to get past bot checks.

• Only one request, the Bechdel Test lookup, names Clusterflick in its user agent.
• Most plain requests go out with Node's default user agent, node. That's honest about being a script, but it doesn't say whose.
• Some plain requests send a Firefox user agent instead. We added it where a site refused node after a burst of requests but served the same request carrying a browser's user agent.
• Every headless-browser fetch runs Playwright with the stealth plugin, which hides the usual signs of automation, so those requests look like an ordinary Chrome visitor.
• Camoufox, a browser built to pass bot checks, is mostly an escalation: we try a plain request first, and switch when the site serves a bot challenge.

Where we clash

At 15 places, what we do goes against the site's own published rules. robots.txt disallows the pages we fetch at 4, and the terms ban automated access or extraction at 12 (Rotten Tomatoes in both).

3 more are borderline: Luma, WeGotTickets and Curzon Sea Containers. Their terms limit how the site may be accessed without naming scraping, or ban it on a site whose reach over the pages we read isn't clear.

Another 10 say nothing about automated access but limit copying, republishing or commercial use of their content. We show their listings, so that's arguable too, and they're marked in yellow rather than counted as clashes.

What stands out

1. robots.txt and terms usually disagree. 10 sites allow every page we fetch in robots.txt and ban automated access in their terms, among them Eventbrite, Letterboxd, Vue and Curzon.
2. The big platforms forbid it; the small venues mostly say nothing. Of the 42 venue sites we read directly, 15 link no terms from their homepage and 25 have terms with nothing on automated access.
3. Some sites turn scripts away from their own rules. 11 terms pages or robots.txt files refused automated readers, or needed JavaScript to show anything. Metacritic's terms answer scripts with HTTP 402 Payment Required. A script that wanted to check the rules first couldn't, so these were read in a real browser.
4. We hold an account at one source. Cinebur's listings can only be read signed in, so we read them through a Clusterflick account. Its sign-up form says that signing up means agreeing to Cinebur's terms, with nothing to tick, but links to a page that returns 404. The terms linked from its website say they cover the app, and exclude “data mining, robots, or similar data gathering and extraction tools”.
5. “Non-commercial” is never defined. TMDB's API terms, IMDb's dataset and the Bechdel Test's licence all allow only non-commercial use, and none says what that means.

Film data and ratings6

These attach each film's details and its scores to the screenings.

Ticketing and listings platforms30

323 of our 488 venues have no listings of their own that we read. Their screenings come only from these platforms, where we search for film events in London and match each to a venue.

Cinema chains and shared booking systems23

120 venues are read through one shared module per chain, or per booking system that several independents use.

Venues' own websites42

42 venues are read from their own websites. Their terms were found from the homepage and searched for clauses on scraping, robots, crawling, automated access, data mining and extraction. That will miss a clause worded some other way, and a terms page not linked from the homepage.

How this was checked

• The list of places comes from Clusterflick's scripts as they stood on 30 October 2026: every venue, every listings platform, the shared chain modules and the film lookups. 3 venues whose listings we type up ourselves are left out, as nothing is fetched for them.
• robots.txt was fetched once per site, identifying as Clusterflick, and every path we request was checked against its rules for all user agents, since we don't send a crawler name a site could address directly.
• Terms were found from each site's homepage and searched for clauses on automated access. For the film data sources, platforms and chains they were then read and quoted. For venues' own sites, only the search was done.
• Pages that refused scripts or needed JavaScript were opened one at a time in a normal Chrome browser, which Claude drove on our behalf. No script was ever retried with a disguised user agent.
• Not covered: the occasional scripts we use to check our coverage and find new venues, and the one we use to fetch venue logos and photos.

Quotes are short extracts: follow the links for the full terms. Like our datasets, this page is free to use with credit.