Parser
Tiny Go library to fetch webpages, extract metadata/content, and tokenize text.
Every scraping project I start ends up reimplementing the same fetch-parse-extract boilerplate. So I packaged it into a lightweight Go library for fetching web pages and extracting their content and metadata. It fetches a URL, parses the HTML, and extracts metadata like title, description, and Open Graph tags, along with the main text content. Links are normalized to absolute URLs and noise like script tags, style tags, mailto, and tel links are stripped out automatically.
The library is built on top of goquery for CSS-selector-based HTML parsing with a straightforward pipeline: fetch the page, parse the DOM, extract metadata from the head, strip non-content elements from the body, and return clean structured data. Text tokenization is handled internally with basic string splitting and cleaning.