Sample: CarWale On-Road Prices.csv
One of the first customers who purchased India Car Database was also interested in On-Road prices. This was a very difficult project I done in January 2016, involved about 20 hours of coding in VB.NET to make an application sending javascript requests to Carwale website.
I charged customer $300 of which $200 paid to my programmer and $100 fee to run the scraper… unfortunately I had to spend whole month (running 12 hours per day except the days I was not home) to get price of each car in each city, application works at a rate of 2 requests per second, so 3100 cars × 510 cities = 1632000 seconds = 226 hours, to get all on-road prices, RTO tax and insurance by city.
In April 2016 the customer came back asking for an update. He was not happy that scraping takes 1 month and I was not happy either, so I proposed him to scrap only cars currently in production (because prices of discontinued cars won’t change). This took about 9 days.
In July 2016 the customer asked another update. He suggested scraping only 50 most important cities so the job took only 3 days.
Asking about last update or how often I update: I can provide you full 510 cities x 3100 cars with prices from January 2016 plus partial updates from April and July. Or you can suggest your own list of cities to be scraped and I will scrap only them at negotiable price and update frequency.
If in 2016 number of customers were low, but in the next years the number of customers buying ready-made databases or enquiring service to create new databases increased, so I CANNOT do large scraping jobs that have very low sale volume.
Alternatively I can give you the scraping application and cars URL. You input the URL in application and get on-road prices, you can update as often you want, you only need a Windows computer running many hours per day (24/7 ideally).
Goods and Services Tax was implemented on 1 July 2017 and harmonized car prices across India, the initial customer told me that updates are no longer required. Surprisingly, few months later a second customer said that need on-road prices.
In November 2017 Carwale website removed IDs from URL of each car, making scraping application obsolete. While I can ask my programmer to make a new scraping application, this will not allow to scrap on-road prices of discontinued cars because I do not know their URLs, but only current production cars (about 1100 model versions) instead of all cars produced from 1990 to present (4000+ model versions), but low number of people interested in on-road prices made me to abandon this project.
