Plan for Building a Reliable, Stable, Ultra-Fast E-Commerce Data Scraping System -- 2

Заказчик: AI | Опубликовано: 29.12.2025

Dear Freelancer, As a new startup, we need a professional plan for a web scraping system that can handle high-volume product data extraction from e-commerce sites. We've tried PC-based methods but faced limitations. We're looking for a cloud-based solution that's automated, stable, and low-cost. Please provide a detailed plan only for now. The plan which fix our current scenario and implementable for us to fulfill our requirements will be award this project. Our team will look into this plan and discuss within our team for few days. If your plan fits all over as per our requirements, we'll hire you for a separate new project to build it. Current Scenario: 1) We have already used method - scraping using Python installed on PC, before this also tried scraping through Eclipse software on PC. 2) Python installed on PC worked but it is slow. It extracted 5000 URLs with 18 XPath locations each URL in around 4 hours. 3) This PC Python system input sample spreadsheet, output sample spreadsheet and Python scraping code files also attached. 4) Our earlier scraping is unacceptable to achieve our result using PC based system due to very high configuration needed and dependency on person to turn on PC, checking PC scraping time to time for internet connection, electricity downtime problem etc. So, we want to use hurdle free process / system to achieve our objective, for example:- online / cloud based system (for example:- GitHub action or anything else), also fix all hurdles while scraping to keep scraping workflow smooth and automation to get our final usable data with almost forever stability. Our Requirements: 1) Our requirement is to have system of scraping 1,00,000+ product URLs per run, extracting 30 fields (located through identifiers, for example XPath) per URL (e.g., rating, price, title, images URL, categories, stock, delivery info, product tags, even minor or major details of products etc) in ~1 hour or less in starting. 2) We want to use hurdle free process / system to achieve our objective, for example:- online / cloud based system (for example:- GitHub action or anything else), also fix all hurdles while scraping to keep scraping workflow smooth and automation to get our final usable data with almost forever stability. 3) Current we are scraping data from Amazon India and Flipkart but further we will also scrap data from our websites as well. 4) Also, as we are very new startup with further plans we do not want to keep high or medium fixed expense per month on our head. We need to run this program daily, twice a day with very very low monthly expense. 5) Automation: Scheduled (daily twice) or manual trigger, no local PC needed, also needed for binding some points of workflow. 6) Scalable: Easy to add new sites (config for XPaths per site). 7) Easy to use: Beginner-friendly daily operation with minimal maintenance. Please Bid with Answering Below Questions: 1) Monthly fixed cost we need to spend on these resources? 2) Time will taken daily by your system for scraping 1,00,000 URLs (with 30 specified elements to scrap in each single URL into different cells of same row for same product, see sample output spreadsheet for more clarity). 3) Overview of techniques, ways which will be used by you to make this workflow system. Anything you want to convey like limitations, important points or anything about your system, if any. 4) Your plan PDF. Next Step: Reply with your plan. Top plans will be selected for the development project.