Instruction file imported from SquidRings1/graph_rag (.github/instructions/extraction.instructions.md). Copyright stays with the author.

Extraction — turning a website or article into graph data

This applies whenever you're working under data/: fetching a source and/or writing data/*.md + data/*.entities.json pairs.

Ingesting a website

When asked to pull information from a URL (e.g. "extract this: https://example.com/article"):

1. Fetch the page and pull out the substantive article/body text — strip nav, ads, comments, and other boilerplate.
2. Derive a filename slug from the page title (kebab-case, e.g. nyt-openai-lawsuit-ruling.md). If data/<slug>.md already exists, reuse it (you're re-processing the same source) rather than creating a duplicate.
3. Write data/<slug>.md with front-matter and the cleaned body:

   ---
   title: "<page title>"
   source: "<the URL>"
   date: "<published date if you can find one, else today's date>"
   ---
   
   <cleaned article text>

4. Immediately continue into the extraction step below for that same file — don't stop at just saving the Markdown.

Extracting entities & relationships

When asked to process data/<name>.md (including one you just fetched above):

1. Read ontology.json for the allowed entity_types and relation_types.
2. Read the article body (everything after the --- front-matter block).
3. Extract entities and relationships strictly within that ontology.
4. Write data/<name>.entities.json (same stem, .entities.json suffix) matching exactly this shape:

   {
     "entities": [
       {"name": "...", "type": "ONE_OF_ENTITY_TYPES", "description": "..."}
     ],
     "relationships": [
       {"source": "...", "target": "...", "relation": "ONE_OF_RELATION_TYPES", "description": "..."}
     ]
   }

   source/target must exactly match an entity name you extracted (or another article's, if the entity already exists in the graph — check existing *.entities.json files to reuse the same spelling instead of creating a duplicate node for the same real-world thing).
5. See data/example-lawsuit.md + data/example-lawsuit.entities.json for a fully worked example of the input/output shape.

After extracting

Run:

python build_graph.py

This merges every data/*.md + *.entities.json pair, runs Louvain community detection, and writes graph_data.json + ai_copyright_graph.html. Re-run it any time — it's cheap and fully deterministic.

Conventions

• Adding a new entity/relationship type means updating ontology.json first — build_graph.py and every future extraction must stay in sync with it.
• Keep entity/relationship descriptions non-empty and specific; they're the only thing a reader (human or you, later) has to go on besides the bare graph structure.