How to Scrape Hotel Listings from Booking.com using Python and BeautifulSoup?
Category
Hotel & Travel
Publish Date
November 2, 2021
Author
Scraping Intelligence
Scraping hotel listings from numerous websites is one of the most popular Web
Scraping apps. This might be done by keeping an eye on rates, creating an aggregator, or improving
the UX of existing hotel booking platforms.
This is accomplished with the help of a simple script. We’ll use BeautifulSoup
to retrieve information, and we’ll use Booking.com to find hotel information.
To begin, we’ll need to get the Booking.com search results page and set up
BeautifulSoup to enable us query the page for meaningful data using CSS selectors.
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requestsheaders = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'}
url = 'https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB;sid=7101b3fb6caa095b7b974488df1521d2;city=-2109472;from_idr=1&;dr_ps=IDR;ilp=1;d_dcp=1'response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content,'lxml')
We also pass the user agent information to imitate a browser call in order to
avoid being blacklisted.
Let’s take a look at the Booking.com search results for a specific
destination. As you can see, it appears to be this way.
When we examine the page, we notice that each item’s HTML is contained within
a tag with the class sr_property_block.
We could simply use this to divide the Html file into these cards, each of
which has information about a single object, such as this:
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requestsheaders = {'
User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2)
AppleWebKit/601.3.9 (KHTML, like Gecko)
Version/9.0.2 Safari/601.3.9'}
url = 'https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB&sid=eae1a774e77c394c5e69703d37e033a3&sb=1&src=searchresults&src_elem=sb&error_url=https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB;sid=eae1a774e77c394c5e69703d37e033a3;tmpl=searchresults;city=-2109472;
class_interval=1;dest_id=-2109472;
dest_type=city;dr_ps=IDR;dtdisc=0;
from_idr=1;ilp=1;inac=0;index_postcard=0;label_click=undef;
offset=0;postcard=0;room1=A%2CA;sb_price_type=total;shw_aparth=1;slp_r_match=0;
srpvid=7df1609ef03a0103;ss_all=0;ssb=empty;sshis=0;top_ufis=1&;&sr_autoscroll=1&ss=Rishīkesh&is_ski_area=0&ssne=Rishīkesh&ssne_untouched=Rishīkesh&city=-2109472&checkin_year=2020&checkin_month=3&checkin_monthday=4&checkout_year=2020&checkout_month=3&checkout_monthday=5&group_adults=2&group_children=0&no_rooms=1&from_sf=1'response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content,'lxml')
#print(soup.select('.a-carousel-card')[0].get_text())
for item in soup.select('.sr_property_block'):
try:
print('----------------------------------------')
print('----------------------------------------')
except Exception as e:
#raise e
print('')
When you execute it:
python3 scrapeBooking.py
The code is clearly isolating the HTML of the cards.
On closer inspection, you’ll notice that the hotel’s name is always preceded
by the sr-hotel-name-class… While we’re at it, let’s collect the number of reviews, pricing, and
ratings.
for item in soup.select('.sr_property_block'):
try:
print('----------------------------------------')
print(item.select('.sr-hotel__name')[0].get_text().strip())
print(item.select('.hotel_name_link')[0]['href'])
print(item.select('.bui-review-score__badge')[0].get_text().strip())
print(item.select('.bui-review-score__text')[0].get_text().strip())
print(item.select('.bui-review-score__title')[0].get_text().strip())
print(item.select('.hotel_image')[0]['data-highres'])
print(item.select('.bui-price-display__value')[0].get_text().strip())
We also attempted to obtain the hotel image and link, as well as other
critical pieces of information.
This is how the entire code appears.
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requestsheaders = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'}
url = 'https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB&sid=eae1a774e77c394c5e69703d37e033a3&sb=1&src=searchresults&src_elem=sb&error_url=https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4AQfIAQzYAQHoAQH4AQKIAgGoAgO4AvTIm_IFwAIB;
sid=eae1a774e77c394c5e69703d37e033a3;tmpl=searchresults;
city=-2109472;class_interval=1;
dest_id=-2109472;dest_type=city;
dr_ps=IDR;dtdisc=0;
from_idr=1;ilp=1;inac=0;index_postcard=0;label_click=undef;
offset=0;postcard=0;room1=A%2CA;sb_price_type=total;
shw_aparth=1;slp_r_match=0;srpvid=7df1609ef03a0103;
ss_all=0;ssb=empty;sshis=0;top_ufis=1&;&sr_autoscroll=1&ss=Rishīkesh&is_ski_area=0&ssne=Rishīkesh&ssne_untouched=Rishīkesh&city=-2109472&checkin_year=2020&checkin_month=3&checkin_monthday=4&checkout_year=2020&checkout_month=3&checkout_monthday=5&group_adults=2&group_children=0&no_rooms=1&from_sf=1'response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content,'lxml')#print(soup.select('.a-carousel-card')[0].get_text())for item in soup.select('.sr_property_block'):
try:
print('----------------------------------------')
print(item.select('.sr-hotel__name')[0].get_text().strip())
print(item.select('.hotel_name_link')[0]['href'])
print(item.select('.bui-review-score__badge')[0].get_text().strip())
print(item.select('.bui-review-score__text')[0].get_text().strip())
print(item.select('.bui-review-score__title')[0].get_text().strip())
print(item.select('.hotel_image')[0]['data-highres'])
print(item.select('.bui-price-display__value')[0].get_text().strip())
print('----------------------------------------')
except Exception as e:
#raise e
print('')
When it is executed:
fetches all the scraped Booking.com hotel data
In more complicated solutions, you’ll need to rotate the User-Agent string so
Booking.com doesn’t recognize the browser!
If we go a step further, we’ll see that Booking.com can just block your IP,
ignoring all of your previous attempts. This is a disappointment because it’s where the majority of
web crawling.
programs fall short.
Overcoming IP Blocks
Buying a private rotating proxy service like Proxies API can often mean the
difference between a successful and pain-free web scraping operation that regularly gets the job
done and one that never does.
Plus, with the current offer of 1000 free API requests, there’s absolutely
nothing to lose by using our revolving proxy and comparing notes. It simply takes one line of
integration to make it almost unnoticeable.
Our rotating proxy server Proxies API is a straightforward API that instantly solves any IP
Blocking issues.
There are millions of high-speed spinning proxies scattered over the globe.
With our IP rotation service, you can rest assured that your IP address will be changed
Hundreds of our customers have successfully solved the problem of IP blocks with a simple API using our automatic User-Agent-String rotation (which simulates requests from different, valid web
browsers and web browser versions) and our automatic CAPTCHA cracking technology.
In any programming language, a basic API like the one below can be used to access the entire system.
Scraping Intelligence Editorial Team is a collective of data specialists, analysts, and researchers with expertise in web scraping, data extraction, and market intelligence. The team produces well-researched guides, actionable insights, and industry-focused resources that help businesses unlock the value of data and make informed, strategic decisions.
Latest Blog
Explore our latest content pieces for every industry and audience seeking information about data scraping and advanced tools.
Learn how to find job postings across multiple websites automatically, explore four common approaches to job data collection, and understand how to set up an automated workflow.
