Custom Multi-Source Aggregator

Замовник: AI | Опубліковано: 26.03.2026

I need a developer to build a fully-functional aggregator that can collect information from Websites, APIs and RSS feeds, normalise the data, and make it available through a single, well-documented endpoint or database. Scope of work • Set up reliable crawlers/scrapers for selected websites, incorporate any available public or private APIs, and subscribe to RSS feeds. • Implement scheduling, rate-limiting and error-handling so each source is polled responsibly without missing updates. • Store the aggregated output in a structured format (SQL, NoSQL or flat-file; I’m open to your recommendation) with clear metadata for source, timestamp and content type. • Expose the data via REST or GraphQL, and include a minimal dashboard or logfile that lets me monitor fetch status, failures and new items in real time. • Deliver clean, well-commented code plus a short README explaining how to add new sources and deploy the solution on a typical cloud VPS. Acceptance criteria 1. All configured websites, APIs and RSS feeds are queried automatically on the defined schedule. 2. Duplicate entries are detected and suppressed. 3. A single JSON response returns merged results in under one second for a 100-item request on standard hosting. 4. Setup instructions let me reproduce the build from scratch in less than 30 minutes. Feel free to choose the tech stack—Python (Scrapy, BeautifulSoup, FastAPI), Node.js (Cheerio, Axios, Express) or something comparably mainstream—as long as you specify versions and any open-source libraries you use. Please include a brief outline of your approach, the tools you prefer, and an estimated timeline.