oDCM - Web Data for Dummies (Tutorial)

We're about to start with today's tutorial (“web data for dummies”).

• If you haven't done so, open this slide deck at (course material –> week 2) & corresponding Jupyter Notebook
• Coaching sessions start after today's tutorial (with Roshini) + finalize team assignment on Canvas
• You can work in Google Colab today

• Today's lecture
  • Work through the in-class tutorial (“webdata for dummies in class.ipynb”)
  • We'll work on a selection of exercises on web scraping and APIs
  • Address some data selection challenges highlighted in “Fields of Gold”
    
• After class
  • Check out today's workplan
  • Complete in-class tutorial and exercises; solutions available online

Use your smartphone to indicate your status.

• Face/screen up: DONE :)
• Face down: still working.

Thanks.

• official vs. unofficial data access
• scaling (APIs scale more)
• web scraping largely free, APIs for-pay
• APIs are linchpin of internet economy

Source & more details: Web Appendix of “Fields of Gold”

• To do scraping, we need to understand what websites consist of:
  • HTML (“what you see and where?”)
  • CSS (“how it looks”)
  • Javascript (“how it interacts”)
    
• Examples:
  • toybox at codepen: https://codepen.io/rcyou/pen/QEObEk/
  • inspecting tilburguniversity.edu

• We need to decide which information to extract from a site
  • is the information publicly accessible or hidden after a login wall?
  • can we reliably get the data, even after many iterations on the site?
  • which information do we need to justify that what we measure is measured well (“construct operationalization”)?

Example: Suppose you need to get… data on Spotify's streaming charts - where would you get it from?

• Best practices
  • Explore different types of pages and providers
  • Get “used” to the website, browse a bit (“how to navigate”), become a customer
  • Identify roadblocks such as captchas
  • Explore limits to iterating through a site (e.g., max. 1000 pages)

• Open music-to-scrape.org
• Familiarize yourself with the structure of the site
• Check which data you encounter
• Suppose the data was stored in a database, how would you structure it?

• But… we don't have access to the database. All we see is the website.
• So, let's “narrow” down on the information of interest
• For this, we can use various extraction techniques on the HTML source code of the site
  • tags, e.g., <table>, <h1>, <div>
  • attributes, e.g., id <table id="example-table">
  • “special” attributes such as classes, e.g., <table class="striped-table">
  • attribute-value pairs, e.g., <table some_field_name = "123">

Frequently, you need to combine several of these methods to extract information.

• Open https://music-to-scrape.org/artist?artist-id=ARICCN811C8A41750F
• How could you uniquely capture the following information?
  • name of the artist
  • number of song plays on the platform
  • top 10 songs?

• When following tutorials on the web, some coders advocate for the use of CSS selectors and XPATHS to extract information.
  
• XPATH
  • Think of it as the “path” to the specific data point
  • Example for “artist name”: /html/body/div[3]/section[1]/div/div[2]/h2
  • Likely to break very easily (say, something changes before the particular element)

• CSS selector
  • Example for extracting “artist name”: .artist_info_title
  • Works here, but can be “highly” dependent: say, tag & class name + position (sometimes too detailed) & can also break
    
• So, here, we are sticking mostly to classes, attributes, and attribute-value pairs.

• We will use three libraries:
  • requests: downloads data from the web or transmits data (“headless”)
  • BeautifulSoup: structures HTML data so we can query it
  • selenium + chromedriver: simulates a browser (chrome), can scroll, click, view the site, but can also be headless; also structures HTML so we can query it
  • json: structure and query JSON data

• For basic, “static” websites, we use requests + beautifulSoup (speed + ease!)
• For dynamic websites, we use selenium (advanced tutorial), OR: selenium (retrieve data) + beautifulSoup (structure/query)
• For APIs, we exclusively use requests
• For data storage, we use json (preferred) or CSV files (flat files with rows & columns)

- Today: requests + BeautifulSoup

• So far, we've just 'lump-extracted' all of the text

• Next, let us refine our collection by getting…

  • the exact location (–> stored in location), exact number plays (–> stored in plays), and the total number of songs in the top 10.

• Tips

  • .find(class_='class-name') for classes
  • you may need to use .find_all()
  • len() for counting
  • remember to write code like “an onion”

location = soup.find(class_ = 'about_artist').find_all('p')[0].get_text()

plays = soup.find(class_ = 'about_artist').find_all('p')[1].get_text()

song_table = soup.find(class_ = 'top-songs')

number_of_songs = len(song_table.find_all('tr'))-1

• JSON is most flexible format, supports “hierarchical data”
• Demonstrate how to create object
• “Full” JSON objects, vs. new-line separated JSON objects
  • Full: the entire FILE (e.g., data.json) is ONE giant JSON object
  • New line separation: each line in your file has one JSON object

Change the code so it stores artist name, location and plays in the JSON object.

def download_data(url):
  web_request = BeautifulSoup(requests.get(url).text)
  artist_name = soup.find('h2').get_text()
  location = soup.find(class_ = 'about_artist').find_all('p')[0].get_text()
  plays = soup.find(class_ = 'about_artist').find_all('p')[1].get_text()

  out_data = {'artist': 'artist name',
            'location': 'store location here',
            'plays': 'store plays here'}

  return(out_data)

The final step is to save JSON data in a file.

We can do this with the JSON library.

import json

out_data = {'artist': 'artist name',
          'location': 'store location here',
          'plays': 'store plays here'}

to_json = json.dumps(out_data)

f=open('filename.json','a')
f.write(to_json+'\n')
f.close()

• In the demos above, we just used a hard-coded list of URLs
• In practice, getting to that list of books/products/songs/artists/users (generally speaking: “seeds”) is a web scraper in itself!
• Examples
  • scrape all artists featured on the homepage
  • then loop through them an extract information for each artist (e.g., total number of)
  • scrape user names from reddit.com; then, use the reddit API to get user metadata
  • …

• Many challenges in sampling
  • sample size? generalizability? panel attrition?
  • all of the firm's data? or just a little bit? are subjects vulnerable (say, kids)?
  • can we actually “get” data on all subjects? can we match data?
• Some solutions and best practices discussed in “Fields of Gold”
• Let's look at the tables in the paper and try to find out…

• Have list of seeds to start up data collection (here, artist IDs)
• loop through list of URLs
  • store raw data for diagnostic purposes
  • extract relevant data to JSON file with raw data
• schedule
• infrastructure

• Get content from a website
• Since we don't want “everything”, we need to “find” relevant elements on the site using
  • classes
  • attributes
  • (attribute-value pairs)
• We first build a prototype; then gradually improve it by
  • modularizing code as much as possible (using functions and loops)
  • ensuring code runs top-down

• Standard way for exchanging data, functions or algorithms
• Which APIs did you already encounter/explore?
• Music to scrape is super easy to use - other APIs require advanced authentication procedures
• Structure corresponds to web scraping
  • have list of seeds (e.g., artist IDs; this was “URLs” for web scraping earlier)
  • store data in new-line separated JSON files (if necessary: convert back to CSV)

• Let's view the documentation

• Let's explore a first endpoint: https://api.music-to-scrape.org/artists/featured

–> we can use the browser to retrieve data from (simple) APIs

Extensions:

• time data retrieval (e.g., every five seconds; discuss multiple strategies)
• store featured artists in a list (or file)

Get some artist meta data for an artist of your choice!

1.) Find an artist IDs on the site or in previous code 2.) Try to make a web request (in your browser) to the following URL:

• Endpoint: https://api.music-to-scrape.org/artist/info
• Requires parameter: artistid
• Combined URL: https://api.music-to-scrape.org/artist/info?artistid={ENTER ARTIST ID HERE}

3.) Does it work? Then write Python code to retrieve the data.

4.) Finally, wrap your code in a function so you can later retrieve data for multiple artists.

• Here, we extracted data every few seconds.
• But, extraction frequency vastly differs by project
• Considerations
  • archival vs. live data?
  • at which frequency does your phenomenon occur?
  • what's the refresh rate of the data source?
  • any excessive burden on server's caused by frequency of extraction?

• Some solutions
  • explore gains of live data collections
  • adhere to best practices (say, 1 req per second)
  • randomize extraction order
  • use automatic schedulers for consistency

• Processing can have various degrees
  • just “save” the raw data
  • extract only necessary information
  • choose data format for saving (say, JSON vs. CSV)
• Some selected challenges
  • GDPR vs value in retaining raw data
  • Anonymization or pseudonymization required?

• Solutions
  • Retain raw when possible
  • Parse minimal amount of data on the fly
  • Remove sensitive info
  • Ensure proper encoding

Please work on exercise 2.4 (see tutorial).

1. collect (longer) list of featured artists
2. for each of these featured artists, collect meta data
3. store data in new-line separated JSON file

• Work on your team project (see workplan on the course website) + make team allocation definite
• Complete this tutorial at home
• Get engaged - see the “bigger” picture of web scraping and APIs
  • discuss ideas & business opportunities (e.g., OpenAI's developer platform?)
  • use our course chatbot at odcm.tilburgai.nl
  • be in touch on WhatsApp for any issues/bugs/etc.