π» Week 03, Day 01 - Lab
Scrape and Parse HTML
By the end of this lab, you should be able to: i) Fetch a web page with requests and parse it with BeautifulSoup, ii) Inspect the HTML to find the right table and understand its structure, iii) Extract headers and data rows into a pandas DataFrame.
β° Monday, 27 July 2026 | Check your timetable for your class time π Check your timetable for the location of your class
This morning you saw how BeautifulSoup turns raw HTML into a tree you can navigate. This afternoon you apply the same tools to a different Wikipedia page: the List of London Underground stations. Your class teacher will walk you through the notebook step by step.
If your final project does not involve web scraping, it is completely fine to use this time on your project instead. Your class teacher can help you if he manages to find a bit of time!
π£οΈ Lab Roadmap
Open ME204_W03D01_Lab.ipynb in VS Code on Nuvolos. All code is given except one exercise in Part III.
| Part | Time | What you do |
|---|---|---|
| Parts I & II | ~20 min | Guided: fetch the page, parse the HTML, find the stations table |
| Part III | ~30 min | Guided buildup, then you write a list comprehension to extract the headers |
| Part IV | ~20-30 min | Guided: discover why a column is missing, extract all rows into a DataFrame |
| Parts V-VI & Bonus | Remaining | Run through the saving step, or work on your final project |
Parts I & II: Fetch, parse, and find the table
Your class teacher will work through this part with the whole room. Run each cell in the notebook and follow along.
This section recaps the web scraping pattern from this morningβs lecture, applied to a new page. The key difference from an API: response.text returns raw HTML (one long string of tags), not JSON. BeautifulSoup parses that string into a tree you can search with find and find_all.
By the end of Part II you will have all the table rows stored in a variable called rows.
Part III: Extract the header row
Your class teacher will walk you through the buildup: look at one row, inspect its cells, extract one value with .text.strip().
Then it is your turn: write a list comprehension that collects the text from all header cells into a list.
Stuck?
Start with afor loop that appends each headerβs text to a list. Once it works, rewrite it as a list comprehension. It is fine to ask an AI to help you translate the loop into a list comprehension. Figuring out what to do is more important than the syntax.
Part IV: Extract the data rows
Your class teacher will work through this part with the whole room. All code is given. Run each cell and follow the discussion.
You will discover that the obvious approach (grab all <td> cells) gives you 10 values when there should be 11. The notebook walks you through finding the missing column and building the correct extraction.
Parts V-VI and Bonus
Run through the remaining sections at your own pace, or switch to your final project.
The Bonus section shows pd.read_html, a shortcut that works when data is inside a clean HTML table. Parts III-V taught you the long route because not all web data is that tidy.
Submit your final project by 5pm on Friday 31 July.
π Appendix
What you need
ME204_W03D01_Lab.ipynb(starter notebook, on Nuvolos)
Data sources
- Wikipedia: List of London Underground stations, public page, educational use.