Custom GeM Portal Data Extraction & Pricing Intelligence Tool 1. Project Overview We are active sellers on the Government e-Marketplace (GeM). We require a robust data extraction tool (web scraper) to perform market research and competitive pricing analysis. The tool should fetch real-time data from specific product categories and catalogues on www.gem.gov.in and export the findings into a structured Excel (.xlsx) format. 2. Technical Requirements The developer must build a solution that can navigate GeM’s dynamic architecture. Key technical challenges include: Dynamic Content: Handling JavaScript-rendered pages and "Load More" buttons. Anti-Bot Measures: Handling CAPTCHAs or session timeouts if they occur during deep-link fetching. Deep Crawling: The tool must go beyond the search results to fetch "Background Details" (specific seller terms, quantity-based discounts, and delivery timelines). 3. Data Points to be Fetched (Mandatory) For every product/catalogue identified, the tool must capture: Product Info: Category, Catalogue ID, Brand, and Model Number. Market Depth: Number of sellers competing on the specific catalogue. Seller Details: Registered location (State/City) and Delivery Lead Time (No. of days). Pricing Intelligence: * Unit price offered by each individual seller. Minimum Order Quantity (MOQ) for that price. Quantity-Based Discounts: Step-pricing (e.g., price change at 10, 50, or 100 units). Inventory: Total available stock quantity per supplier. 4. Deliverables The Executable/Script: A Python script (preferred) or a standalone desktop tool. Excel Output: A clean .xlsx file with headers for all the data points mentioned above. Documentation: A brief "How-to" guide on running the tool and updating search parameters (e.g., how to change the Category URL or Keywords). 5. Preferred Freelancer Skills Expertise in Python (Selenium, BeautifulSoup, or Scrapy). Experience with E-commerce scraping and bypassing bot detection. Previous experience with Indian Government Portals (GeM, CPP, or Tenders) is a major plus. Ability to deliver a tool that does not get our IP blocked (rate-limiting features). Important Note on Compliance Since GeM is a government portal, ensure your developer follows ethical scraping practices (e.g., adding delays between requests) to avoid putting undue load on the portal servers, which could lead to your account or IP being restricted. The "Deep Scrape" Requirement: look for the "Seller Wise Price" tab. GeM often hides the full list of sellers and their specific quantity discounts under a secondary click or a pop-up window. State-Specific Filtering: GeM prices change based on the Consignee Location so need a "Location Selector" so we can see prices offered specifically for your target states (e.g., Maharashtra vs. Karnataka). Timestamping: Will need a "Fetch Date" column, as GeM prices and stock levels can change multiple times a day.