Imported from TheGraphicKing/skott-canvas (
skills/competitor-scraper/SKILL.md). Install upstream withnpx skills add TheGraphicKing/skott-canvas --skill competitor-scraper. Copyright stays with the author.
Overview
The competitor-scraper skill fetches public web content from six competitor source types and returns structured, deduplicated artifact payloads. It operates as the foundational data-collection layer of the competitor intelligence pipeline. Every artifact produced by this skill carries full provenance metadata so downstream classifiers, theme extractors, and gap-analysis agents can trace each signal back to its origin without re-fetching the source.
The skill is designed around precise structured extraction, not raw HTML forwarding. It parses each source type with purpose-built extraction logic and returns only the fields downstream agents need — saving token budget, avoiding hallucination risk in downstream agents, and enabling robust deduplication.
When to Use
- When the competitor-scheduler dispatches a scan job for a specific competitor
- When a one-off competitive intelligence request requires fresh data from a specific source type
- When the gap-analysis-agent needs a spot-check refresh of a competitor's pricing or careers page
- When the weekly-digest-generator needs a manual pull of recent press mentions before digest assembly
How to Execute
Step 1: Job Validation
Validate the incoming job payload: confirm that competitor ID, source URL list, run ID, and last-scan timestamp are all present. If any required field is missing, return an error and do not proceed. Load the competitor's entry from the competitor-intel knowledge file to confirm the source URL list is current — the knowledge file is the source of truth, not the job payload.
Step 2: Blog Scraping
Fetch the competitor's blog or news index. Parse all posts with a publication date after the last-scan timestamp. For each post extract:
- title (string)
- url (string)
- published_at (ISO 8601 date)
- author (string, if available)
- word_count (integer)
- body_text (string, full text)
- topic_tags (array of strings)
- product_claims (array: any named features, capabilities, or integrations mentioned)
- customer_references (array: any named customers, logos, or case study references)
- data_citations (array: any third-party data, analyst reports, or statistics cited with source)
Limit to 20 most recent posts unless the scheduler has set a high-activity override.
Step 3: LinkedIn Post Scraping
Retrieve recent public LinkedIn company posts. For each post extract:
- post_text (string)
- published_at (ISO 8601 date)
- post_type (enum: organic, product-announcement, event-promo, hiring, customer-story, thought-leadership)
- engagement_signals (object: likes, comments, reposts — include only if visible without authentication)
- external_links (array: any URLs linked from the post)
- lyzr_mention (boolean: does the post name Lyzr or a Lyzr-adjacent term?)
Step 4: Pricing Page Scraping
Fetch the competitor's pricing page. Extract:
- pricing_model (enum: per-seat, consumption, flat-rate, enterprise-only, freemium)
- tier_names (array of strings)
- tier_descriptions (array: one string per tier)
- feature_gates (array: features listed as tier-specific)
- deployment_options (array: cloud, on-prem, hybrid — only if explicitly stated)
- cta_language (string: the primary call-to-action text, e.g., "Start Free Trial" or "Contact Sales")
- minimum_contract_signals (string: any language implying deal size floor, e.g., "for teams of 50+")
- diff_from_previous (boolean: has this page changed since the last scrape?)
Step 5: Careers Page Scraping
Fetch the competitor's open roles. Aggregate by department. Extract:
- total_open_roles (integer)
- roles_by_department (object: department name → count)
- ai_ml_engineering_count (integer)
- sales_count (integer)
- vertical_specific_roles (array: role titles that name a vertical — BFSI, healthcare, manufacturing)
- net_new_roles_vs_prior (integer: positive = growth, negative = shrinkage)
- notable_senior_roles (array: VP, C-suite, or Director-level roles newly posted)
Step 6: G2 Review Scraping
Retrieve the most recent G2 reviews from the competitor's product listing page. For each review extract:
- star_rating (integer 1-5)
- reviewed_at (ISO 8601 date)
- reviewer_title (string)
- reviewer_company_size (enum: 1-50, 51-200, 201-1000, 1001-5000, 5000+)
- pros_text (string)
- cons_text (string)
- features_praised (array: named product features mentioned positively)
- features_criticized (array: named product features mentioned negatively)
- high_signal_flag (boolean: true if review mentions enterprise deployment, pricing, implementation complexity, or support response time)
Do not extract reviewer names — reviewer title and company size are sufficient for signal analysis.
Step 7: Press Mention Scraping
Search configured press sources and Google News for the competitor name within the scan window. For each mention extract:
- headline (string)
- publication (string)
- published_at (ISO 8601 date)
- url (string)
- summary (string: 2-3 sentence summary of the article's relevance)
- mention_type (enum: product-launch, partnership, funding, analyst-coverage, award, executive-change, negative-coverage, other)
- lyzr_mentioned (boolean: does the article also mention Lyzr or a direct Lyzr competitor alongside this competitor?)
Step 8: Deduplication and Output Assembly
For every extracted artifact, compute a fingerprint: SHA-256 hash of (source_type + source_url + published_at + title). Check the artifact store for the fingerprint. If the fingerprint exists, discard the artifact and increment the duplicate count. Only write net-new artifacts to the output payload.
Each output artifact must include: competitor_id, source_type, artifact_id (UUID), run_id, fingerprint, scraped_at (ISO 8601), and the structured fields for that source type.
Inputs
- job_payload (object, required): Competitor ID, source URL list, run ID, last-scan timestamp
- artifact_store_reference (string, required): Reference to the artifact store for deduplication checks
- high_activity_override (boolean, optional): If true, lift the 20-post limit for blog scraping
Outputs
- artifacts (array): Array of structured artifact objects, one per net-new item discovered
- duplicate_count (integer): Number of artifacts discarded as duplicates
- source_errors (array): List of source URLs that returned errors, with status codes
- job_completion_status (enum: complete, partial, failed)
Quality Criteria
- Every artifact must include all required metadata fields — an artifact with a missing run_id or fingerprint is invalid and must be discarded
- No fabricated or inferred fields — if a field cannot be extracted directly from the page content, it must be null, not estimated
- Deduplication must run before output assembly — duplicate artifacts must never appear in the output array
- Source errors must be reported accurately — a 429 rate-limit error and a 404 not-found error require different handling by the scheduler and must be reported with the correct status code