Scraping hotel listings from numerous websites is one of the most popular Web Scraping apps. This might be done by keeping an eye on rates, creating an aggregator, or improving the UX of existing hotel booking platforms.
This is accomplished with the help of a simple script. We’ll use BeautifulSoup to retrieve information, and we’ll use Booking.com to find hotel information.
To begin, we’ll need to get the Booking.com search results page and set up BeautifulSoup to enable us query the page for meaningful data using CSS selectors.
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requestsheaders = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'}
url = 'https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB;sid=7101b3fb6caa095b7b974488df1521d2;city=-2109472;from_idr=1&;dr_ps=IDR;ilp=1;d_dcp=1'response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content,'lxml')
We also pass the user agent information to imitate a browser call in order to avoid being blacklisted.
Let’s take a look at the Booking.com search results for a specific destination. As you can see, it appears to be this way.
When we examine the page, we notice that each item’s HTML is contained within a tag with the class sr_property_block.
We could simply use this to divide the Html file into these cards, each of which has information about a single object, such as this:
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requestsheaders = {'
User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2)
AppleWebKit/601.3.9 (KHTML, like Gecko)
Version/9.0.2 Safari/601.3.9'}
url = 'https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB&sid=eae1a774e77c394c5e69703d37e033a3&sb=1&src=searchresults&src_elem=sb&error_url=https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB;sid=eae1a774e77c394c5e69703d37e033a3;tmpl=searchresults;city=-2109472;
class_interval=1;dest_id=-2109472;
dest_type=city;dr_ps=IDR;dtdisc=0;
from_idr=1;ilp=1;inac=0;index_postcard=0;label_click=undef;
offset=0;postcard=0;room1=A%2CA;sb_price_type=total;shw_aparth=1;slp_r_match=0;
srpvid=7df1609ef03a0103;ss_all=0;ssb=empty;sshis=0;top_ufis=1&;&sr_autoscroll=1&ss=Rishīkesh&is_ski_area=0&ssne=Rishīkesh&ssne_untouched=Rishīkesh&city=-2109472&checkin_year=2020&checkin_month=3&checkin_monthday=4&checkout_year=2020&checkout_month=3&checkout_monthday=5&group_adults=2&group_children=0&no_rooms=1&from_sf=1'response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content,'lxml')
#print(soup.select('.a-carousel-card')[0].get_text())
for item in soup.select('.sr_property_block'):
try:
print('----------------------------------------')
print('----------------------------------------')
except Exception as e:
#raise e
print('')
When you execute it:
python3 scrapeBooking.py
The code is clearly isolating the HTML of the cards.
On closer inspection, you’ll notice that the hotel’s name is always preceded by the sr-hotel-name-class… While we’re at it, let’s collect the number of reviews, pricing, and ratings.
for item in soup.select('.sr_property_block'):
try:
print('----------------------------------------')
print(item.select('.sr-hotel__name')[0].get_text().strip())
print(item.select('.hotel_name_link')[0]['href'])
print(item.select('.bui-review-score__badge')[0].get_text().strip())
print(item.select('.bui-review-score__text')[0].get_text().strip())
print(item.select('.bui-review-score__title')[0].get_text().strip())
print(item.select('.hotel_image')[0]['data-highres'])
print(item.select('.bui-price-display__value')[0].get_text().strip())
We also attempted to obtain the hotel image and link, as well as other critical pieces of information.
This is how the entire code appears.
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requestsheaders = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'}
url = 'https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB&sid=eae1a774e77c394c5e69703d37e033a3&sb=1&src=searchresults&src_elem=sb&error_url=https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB;
sid=eae1a774e77c394c5e69703d37e033a3;tmpl=searchresults;
city=-2109472;class_interval=1;
dest_id=-2109472;dest_type=city;
dr_ps=IDR;dtdisc=0;
from_idr=1;ilp=1;inac=0;index_postcard=0;label_click=undef;
offset=0;postcard=0;room1=A%2CA;sb_price_type=total;
shw_aparth=1;slp_r_match=0;srpvid=7df1609ef03a0103;
ss_all=0;ssb=empty;sshis=0;top_ufis=1&;&sr_autoscroll=1&ss=Rishīkesh&is_ski_area=0&ssne=Rishīkesh&ssne_untouched=Rishīkesh&city=-2109472&checkin_year=2020&checkin_month=3&checkin_monthday=4&checkout_year=2020&checkout_month=3&checkout_monthday=5&group_adults=2&group_children=0&no_rooms=1&from_sf=1'response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content,'lxml')#print(soup.select('.a-carousel-card')[0].get_text())for item in soup.select('.sr_property_block'):
try:
print('----------------------------------------')
print(item.select('.sr-hotel__name')[0].get_text().strip())
print(item.select('.hotel_name_link')[0]['href'])
print(item.select('.bui-review-score__badge')[0].get_text().strip())
print(item.select('.bui-review-score__text')[0].get_text().strip())
print(item.select('.bui-review-score__title')[0].get_text().strip())
print(item.select('.hotel_image')[0]['data-highres'])
print(item.select('.bui-price-display__value')[0].get_text().strip())
print('----------------------------------------')
except Exception as e:
#raise e
print('')
When it is executed:
fetches all the scraped Booking.com hotel data
In more complicated solutions, you’ll need to rotate the User-Agent string so Booking.com doesn’t recognize the browser!
If we go a step further, we’ll see that Booking.com can just block your IP, ignoring all of your previous attempts. This is a disappointment because it’s where the majority of web crawling
programs fall short.
Buying a private rotating proxy service like Proxies API can often mean the difference between a successful and pain-free web scraping operation that regularly gets the job done and one that never does.
Plus, with the current offer of 1000 free API requests, there’s absolutely nothing to lose by using our revolving proxy and comparing notes. It simply takes one line of integration to make it almost unnoticeable.
Hundreds of our customers have successfully solved the problem of IP blocks with a simple API using our automatic User-Agent-String rotation (which simulates requests from different, valid web
browsers and web browser versions) and our automatic CAPTCHA cracking technology.
In any programming language, a basic API like the one below can be used to access the entire system.
curl http://api.websitescraper.com//?key=API_KEY&url=https://example.com
For scraping hotel listings data from Booking.com contact Scraping Intelligence today!!
