Week 2) Foundations of web scraping
Learning goals
By the end of this week, you should understand:
- how information is organized and delivered through websites;
- how researchers can collect information from static and interactive websites;
- why different websites require different data-collection strategies; and
- how technical choices affect the quality, completeness, and limitations of collected web data.
Preparation before class
- Review the relevant sections of the Python Bootcamp.
- Make sure that you can access JupyterLab at jupyterlab.uvt.nl.
- Bring questions or problems encountered while working through the Python Bootcamp.
Tutorials
Web Scraping 101
This tutorial introduces the basic workflow for collecting data from webpages:
- sending requests;
- inspecting HTML;
- parsing webpages with Beautiful Soup;
- selecting elements and attributes; and
- transforming extracted values into a structured dataset.
Web Scraping Advanced
This tutorial introduces browser automation with Selenium. You will practice:
- simulating selected user interactions;
- scrolling, clicking, and filling in forms;
- working with dynamically generated content; and
- saving collected results as JSON.
The advanced tutorial requires a local Python setup and is not designed for JupyterLab in the cloud.
Connecting scraping to your project
When exploring a possible project source, do not begin with the technical question alone. First identify the research, business, or knowledge opportunity that the data could address. Then use the scraping exercises to investigate whether the source contains the information you need and whether it can be accessed reliably.
After class
- Work through the Web Scraping 101 and Web Scraping Advanced tutorials.
- Begin identifying a possible website or API for your team project.
- Register your project team on Canvas.