scaffold-importer

Prompt file imported from enorm-labs/event-junkie (.github/prompts/scaffold-importer.prompt.md). Fill in {{arguments}} before use. Copyright stays with the author.

Scaffold a New Importer

Add a new venue event importer to events-importer, end to end: enum value, parser + importer classes, fixtures, tests, seed venue file — following ADR-007 and the importers already there.

Arguments: {{arguments}} names the venue and its listing URL (SO36 https://so36.com/programm/). Ask if either is missing.

Important

• git --no-pager.
• Read ADR-007 in full first — architecture, selector strategy, scraping ethics. Read the venue's row in EVENT_DATA_SOURCES.md for platform and quirks; it is not a field mapping — analyse the live site and map it onto DATA_MODEL.md yourself.
• Copy the closest existing importer: JSON / API source — scraper/festsaal/, scraper/neuezukunft/, scraper/madameclaude/ (EventImporter directly, ApiClient, one *ApiScraper.kt); single listing page — scraper/privatclub/ (AbstractSinglePageWebsiteImporter); list + detail — scraper/cassiopeia/, scraper/astra/ (AbstractTwoPageWebsiteImporter). New code looks like the code around it; the shared helpers in scraper/ are used, not reimplemented.

1. Reconnaissance

1. robots.txt: honour any Disallow on the listing and detail paths; disallowed means stop and report.
2. Look for a JSON / API source before committing to HTML (ADR-007 selector priority 1): XHR/fetch calls in the Network tab, WordPress /wp-json/wp/v2/<type>, an RSS feed, ?format=json, an embedded calendar widget's boot endpoint (Festsaal is Wagtail REST, Neue Zukunft an Elfsight widget, Madame Claude WordPress event with ACF). <script type="application/ld+json"> schema.org/MusicEvent is still an HTML fetch, parsed as JSON-LD (Astra, Privatclub). A clean JSON source wins.
3. Save the real listing (and one or two detail pages for list+detail, including an edge case — cancelled, sold out, free, missing date) as the fixture: curl -sSL -A 'EventJunkie/1.0 (+https://github.com/enorm-labs/event-junkie)' '<url>' -o events-importer/src/test/resources/scraper/<venue>/<venue>-overview.html. A 403 from curl is not a blocked source. Cloudflare refuses curl by its fingerprint, whatever the User-Agent, and still serves the importer's JDK client (maaya.de: curl 403, JDK 200). Fetch once with java.net.http.HttpClient and the same User-Agent before calling a source blocked.
4. Classify: JSON / API → ApiClient.fetchJson + a pure JSON scraper; JSON-LD → HtmlFetcher, read with jsonLdEvents() and the schema* readers; server-rendered HTML → Jsoup; JS-rendered with no API → stop and flag (Playwright is not in the project, ADR-007 §3, and adding it is a separate decision).
5. Page pattern: single page, list+detail or paginated. First page only (ADR-007 § Pagination); a venue that truly needs more loops inside importEvents(), never a changed interface.

2. The EventSource value

In scraper/EventSource.kt, one line of KDoc about the venue itself — nothing about the site or the parsing, which lives on the importer. The name becomes source_type and the sourceId prefix:

/** SO36 Berlin – the Oranienstraße club central to Berlin's punk and new-wave history. */
SO36,

3. The scraper/<venue>/ package

• <Venue>OverviewPageScraper.kt — pure, no I/O; a Jsoup Document + base URL in (or the raw JSON String), List<ScrapedEvent> out.
• <Venue>DetailPageScraper.kt — list+detail only; ScrapedEvent? for one page.
• <Venue>WebsiteImporter.kt — the @Component that fetches and wires. JSON in one response: extend AbstractJsonApiImporter(apiClient, "<Venue>", <Venue>ApiScraper()::scrape), overriding requestUrl to build a query on the configured base. The Events Calendar: extend AbstractTecImporter. A WordPress REST collection: page it with walkWpRestPages, never one page of 100. Single-page HTML: extend AbstractSinglePageWebsiteImporter(htmlFetcher, "<Venue>", <Venue>OverviewPageScraper()::scrape), which owns the conditional fetch and the NotModified case; a listing paged by a next link overrides maxListingPages and nextListingPage, and an entry page that only links month pages extends AbstractMonthPagesWebsiteImporter. A loop over pages that keeps the others when one fails uses readEach; to add fields from each event's own page, override enrichFromEventPage with a parser. Never fetch an event page yourself: the hook, or withEventPageOrFlagged in an importer that implements EventImporter directly, owns the catch, the WARN and the detailUnavailable flag that keeps the stored fields when the page fails. A performance listing whose rows share one page per production overrides sharedPages, which reads each page once through enrichFromSharedPages and flags every row of a failed one. List+detail: extend AbstractTwoPageWebsiteImporter, pass <Venue>OverviewPageScraper()::scrape and <Venue>DetailPageScraper()::scrape to its constructor. Override fillGapsFromOverview only where the listing is authoritative for a field, as primary.withGapsFrom(fallback).copy(…); the default fills every gap from the listing. A field the detail page supplies over the listing's own value goes in detailPageOwns, so a failed page keeps the stored one; add ScrapedField.ARTISTS where the page's lineup is finer than the listing's. override val eventSource = EventSource.<VENUE>.

The importer's and scrapers' KDoc is the one home for the source — platform, pages read and why, the traps, why a selector was chosen. Under 20 comment lines on the importer, under 10 per scraper; do not copy the boilerplate an existing scraper still carries (purity, fixture setup, @param document the parsed document — #393 deleted it from 106 files). A defect we could repair is an issue, not KDoc.

What the source does not carry is a record (#715). Every importer file ends with one, and the build fails without it:

val <VENUE>_LIMITATIONS =
    VenueLimitations(
        EventSource.<VENUE>,
        AcceptedLimitation(LimitedAspect.DOORS_TIME, "the site publishes only one time per night"),
    )

One AcceptedLimitation per withheld thing, the reason a property of the site, lowercase, one sentence, no full stop. A venue that publishes everything declares VenueLimitations(EventSource.<VENUE>) — a statement, not an omission. /data-quality-audit reads these; a limitation left in prose is re-reported every run. A single-sound club that names no style declares it here as houseGenre = "Techno", never as a genre constant in the scraper: the import fills it into every music event that names no genre of its own.

4. Shared helpers, selectors, fields

ScrapingExtensions.kt (textAt, attrAt, imgSrcAt, hrefAt, resolveUrl), DateParsingExtensions.kt (parseTime, parseIsoDate, parseIsoTime), WeekdayDateExtensions.kt (inferYearForWeekday; neighbouringMonthOnWeekday, for a full date whose printed weekday names another month), EventTypeMapping.kt (mapEventType, refineConcertVenueType, isFestivalTitle), ArtistNameMapping.kt (isPlaceholderName, isNonArtistName, buildArtistList, extractSupportFromSubtitle), EventFieldMapping.kt (parseEventStatus, orderDoorsBeforeStart, cleanEventTitle, detectFree), JsonLdEvents.kt (jsonLdEvents, schemaDate, schemaTime, schemaImageUrl, schemaSoldOut). A helper two venues need goes into the extension file and ADR-007's utility table, not the venue package.

Selectors in ADR-007's order: JSON-LD > semantic HTML5 (article, time[datetime], headings) > ARIA > data-* > meaningful classes > never positional (div:nth-child(3), .col-md-4). Scope to the narrowest container; :has() for context.

ScrapedEvent (scraper/ScrapedEvent.kt): required title, eventDate, sourceUrl, sourceId = "${EventSource.<VENUE>.sourceIdPrefix}$slug" derived from the canonical URL, never from mutable text. Validate before returning: skip a blank title or an unparseable date with a warning, and parse the rows with rows.mapSkippingFailures(logger, "<Venue> event item") { … } so one malformed event does not abort the import; never a hand-written try/catch.

5. Tests

Under src/test/kotlin/de/norm/events/scraper/<venue>/: <Venue>OverviewPageScraperTest (and DetailPageScraperTest) parse the fixture (Jsoup.parse(html, baseUrl), or the raw string) and assert the count, one fully populated event, and the edge cases. <Venue>WebsiteImporterTest mocks ApiClient or HtmlFetcher with MockK (coEvery { … }) and asserts ImportResult.Success with the right events, validators propagated, NotModified passed through, an empty page handled, eventSource matching. Template: PrivatclubWebsiteImporterTest. JUnit 5 + Kotest + MockK; a fixed Clock where the scraper infers the year.

6. Register the source

Sources are runtime rows, not Flyway (ADR-007). Each venue is one file, http/importer/seed/venues/<venue-slug>.json (#2824): the POST /api/admin/venues body under venue (with district, venueTypes and, where the venue publishes one, capacity), and under sources the POST /api/admin/event-sources body with "sourceType": "<VENUE>" and the listing url, without venueId. The file name is the venue's slug (Astra Kulturhaus → astra-kulturhaus.json); SeedVenueFilesTest fails on a wrong one. Then run scripts/dev-seed-parity.sh: it regenerates http/importer/dev-seed.http, the "Run all" file, with the venue, source and import requests. Never edit that file by hand; CI fails when it drifts.

{
    "venue": { "name": "Astra Kulturhaus", "address": "Revaler Str. 99", "...": "..." },
    "sources": [
        {
            "name": "Astra Kulturhaus",
            "url": "https://www.astra-berlin.de/",
            "sourceType": "ASTRA",
            "enabled": true,
            "importIntervalMinutes": 1440,
            "maxRetries": 3
        }
    ]
}

The venue may already have a file, because a venue we know but do not import gets one with a programmeUrl and "sources": [] (#2766). Look for http/importer/seed/venues/<venue-slug>.json first. If it exists, create no second venue: add the source to its sources list and remove its programmeUrl. The row on a cluster keeps its value, which the site stops showing once the source exists.

The venue's description and descriptionAlt are our own prose, English and German, one sentence each. Take the facts from the venue's own About or history page, not from the listing. Open with what sets the venue apart — the building, its history, who programmes it, a house rule such as smoke-free — then name its formats plainly. No hedged tail: "and the odd concert", "occasional club parties", "gelegentliche Konzerte" fit nearly any club and tell a visitor nothing. A minor format is either named without the hedge or left out. The same holds for the one-line EventSource KDoc. A description already on a cluster changes only through a guarded data migration, as V078 does; the seed alone never reaches it.

And a row in docs/licence-review/RESULTS.tsv, keyed on the same name: scripts/seed-sources.py --enable refuses to switch on a source without a verdict, so a source missing here can be created on a cluster and never import. Read the venue's legal and press pages; an agent records UNCLEAR, and a person confirms PERMITTED or PROHIBITED (docs/licence-review/README.md §3). The same row gives the source its translation licence: scripts/apply-licence-review.py derives it from the description verdict (ADR-027, docs/licence-review/README.md §7).

And the onboarding steps in the pull request, under ## After deploy, which label-pr.yml labels after-deploy: the new-venue block from /post-release, copied as written, so a session that runs it after the release seeds, reviews and enables the source in the order the licence gate needs.

7. Verify

./gradlew :events-importer:test --tests '*<Venue>*' --tests '*ModularityTests'
./gradlew :events-importer:ktlintCheck :events-importer:detekt :events-importer:detektMain

Then /verify, then /importer-smoke against the live site — the per-host throttle applies; do not hammer the venue.

Checklist

• robots.txt checked; JSON/API source looked for first; not a JS SPA without an API
• EventSource value with a one-line venue description
• <venue>/ package: scraper(s) + @Component importer; shared helpers reused; semantic selectors
• sourceId stable and prefixed; events validated before return
• <VENUE>_LIMITATIONS at the foot of the importer, registered in AcceptedLimitations.declarations; ACCEPTED_LIMITATIONS.md regenerated
• Fixtures under src/test/resources/scraper/<venue>/; scraper + importer tests cover happy path, edge cases, NotModified, empty page
• The venue file under http/importer/seed/venues/ holds the source; scripts/dev-seed-parity.sh run; a docs/licence-review/RESULTS.tsv row under the same name
• Venue description read against the venue's own About page; opens with what sets it apart, no "the odd …" tail
• venueTypes from the venue's own site, one or more; capacity only from a figure the venue or its operator publishes
• ktlintCheck, detekt, ModularityTests, new tests green; /verify clean