> For the complete documentation index, see [llms.txt](https://eduapps.gitbook.io/python-intro/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://eduapps.gitbook.io/python-intro/9.-simple-web-crawler.md).

# 9. Simple Web Crawler

The following code crawl the first five pages of the BBC news search result (with the search term "hong kong")

```python
import requests
from bs4 import BeautifulSoup

for i in range(1, 5):
    url = 'https://www.bbc.co.uk/search/more?page=' + str(i) + '&q=hong+kong'

    html_text = requests.get(url).text
    html_data = BeautifulSoup(html_text, "html.parser")

    headline_list = html_data.find_all('h1')

    for headline in headline_list:
        print(headline.find('a').get_text())
```