For example, you want to scrape 100 Flipkart products from every category. But this formula will only extract the top 15 products on a page. Flipkart offers a feature called limitless scrolling, which eliminates the need for pagination (such as?page=2,?page=3) in the URL. If it had this functionality, we would have entered the value into a "while loop" and increased the page values as seen below.
page_count = 0 while page_count < 5: URL = "http://example.com?page=%d" %(page_count) # scraping code... page_count += 1
So, let's go back to limitless scrolling.
The usage of "Ajax" allows any webpage to employ limitless scrolling. However, that ajax request also includes a URL from which the goods on the same page are loaded when scrolling.
To look at that URL.
When you click the link in a new window, you'll find it easily to this, with about 15 to 20 goods on each page.
Analyze the URL; there is indeed a Get parameter called?start=. (Some number)
Certain websites' Ajax URLs may additionally include "JSON" answers. If you locate them, you won't need to scrape them; simply retrieve the JSON response as you would any other JSON API.
If you are looking for web scraping service using Infinite scrolling, contact Scraping Intelligence.
Request for a quote!
Copyright © 2009 - 2022 Scraping Intelligence