9. Simple Web Crawler

Introduction to Python

Search…

1. Introduction

2. Numbers in Python

3. Text in Python

4. Making Comparisons

5. Making Choices

6. Exercise: Paper, Scissors, Stone game

7. Lists

8. Loops

9. Simple Web Crawler

The following code crawl the first five pages of the BBC news search result (with the search term "hong kong")
1
import requests
2
from bs4 import BeautifulSoup
3
​
4
for i in range(1, 5):
5
    url = 'https://www.bbc.co.uk/search/more?page=' + str(i) + '&q=hong+kong'
6
​
7
    html_text = requests.get(url).text
8
    html_data = BeautifulSoup(html_text, "html.parser")
9
​
10
    headline_list = html_data.find_all('h1')
11
​
12
    for headline in headline_list:
13
        print(headline.find('a').get_text())
Copied!

8. Loops

Last modified 2yr ago

Copy link