oDCM - Web Data for Dummies (Tutorial)

We're about to start with today's tutorial (“web data for dummies”).

• If you haven't done so, open this slide deck at https://odcm.hannesdatta.com/docs/modules/week2/webdata-for-dummies/
• Coaching session start after today's tutorial (we will change rooms)
• Team assignment on Canvas

• In-class
  • Work through the in-class tutorial (“webdata for dummies in class.ipynb”)
  • We'll work on a selection of exercises on web scraping and APIs
  • Address some data selection challenges highlighted in “Fields of Gold”
    
• After class
  • Work on team activity #1
  • Complete tutorial and exercises + be in touch for feedback during coaching hours

• official vs. unofficial data access
• scaling (APIs scale more)
• web scraping largely free, APIs for-pay
• APIs are linchpin of internet economy
• Learn about other differences and commonalities in the Web Appendix of “Fields of Gold”.

• To do scraping, we need to understand what websites consist of:
  • HTML (“what you see and where?”)
  • CSS (“how it looks”)
  • Javascript (“how it interacts”)
    
• Examples:
  • toybox at codepen: https://codepen.io/rcyou/pen/QEObEk/
  • inspecting tilburguniversity.edu

• We need to decide which information to extract from a site
  • is the information publicly accessible or hidden after a login wall?
  • can we reliably get the data, even after many iterations on the site?
  • which information do we need to justify that what we measure is measured well (“construct operationalization”)?

Example: Suppose you need to get… data on Spotify's streaming charts - where would you get it from?

• Best practices
  • Explore different types of pages and providers
  • Get “used” to the website, browse a bit (“how to navigate”), become a customer
  • Identify roadblocks such as captchas
  • Explore limits to iterating through a site (e.g., max. 1000 pages)

• Open music-to-scrape.org (fallback option: books.toscrape.com)
• Familiarize yourself with the structure of the site
• Check which data you encounter
• Suppose the data was stored in a database, how would you structure it?

• But… we don't have access to the database. All we see is the website.
• So, let's “narrow” down on the information of interest
• For this, we can use various extraction techniques on the HTML source code of the site
  • tags, e.g., <table>, <h1>, <div>
  • attributes, e.g., id <table id="example-table">
  • “special” attributes such as classes, e.g., <table class="striped-table">
  • attribute-value pairs, e.g., <table some_field_name = "123">

Frequently, you need to combine several of these methods to extract information.

• Open https://music-to-scrape.org/artist?artist-id=ARICCN811C8A41750F
• How could you uniquely capture the following information?
  • name of the artist
  • number of song plays on the platform
  • top 10 songs?

• Open https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
• How would you identify the following information?
  • book title
  • price
  • in-stock availability, and
  • number of stars?

• When following some extra tutorials, some coders advocate for the use of CSS selectors and XPATHS to extract information.
  
• XPATH
  • Think of it as the “path” to the specific data point
  • Example for “artist name”: /html/body/div[3]/section[1]/div/div[2]/h2
  • Likely to break very easily (say, something changes before the particular element)

• CSS selector
  • Example for extracting “artist name”: .artist_info_title
  • Works here, but can be “highly” dependent: say, tag & class name + position – can also break
    
• So, here, we are sticking mostly to classes, attributes, and attribute-value pairs.

• So far, we've just 'lump-extracted' all of the text

• Next, let us refine our collection by getting…

  • the exact location (–> stored in location), exact number plays (–> stored in plays), and the total number of songs in the top 10.

• Tips

  • .find(class_='class-name') for classes
  • you may need to use .find_all()
  • len() for counting
  • remember to write code like “an onion”

location = soup.find(class_ = 'about_artist').find_all('p')[0].get_text()
plays = soup.find(class_ = 'about_artist').find_all('p')[1].get_text()
song_table = soup.find(class_ = 'top-songs')
number_of_songs = len(song_table.find_all('tr'))-1

• JSON is most flexible format, supports “hierarchical data”
• Demonstrate how to create object
• “Full” JSON objects, vs. new-line separated JSON objects
  • Full: the entire FILE (e.g., data.json) is ONE giant JSON object
  • New line separation: each line in your file has one JSON object

Change the code so it stores artist name, location and plays in the JSON object.

def download_data(url):
  web_request = BeautifulSoup(requests.get(url).text)
  artist_name = soup.find('h2').get_text()
  location = soup.find(class_ = 'about_artist').find_all('p')[0].get_text()
  plays = soup.find(class_ = 'about_artist').find_all('p')[1].get_text()

  out_data = {'artist': 'artist name',
            'location': 'store location here',
            'plays': 'store plays here'}

  return(out_data)

The final step is to save JSON data in a file.

We can do this with the JSON library.

import json

out_data = {'artist': 'artist name',
          'location': 'store location here',
          'plays': 'store plays here'}

to_json = json.dumps(out_data)

f=open('filename.json','a')
f.write(to_json+'\n')
f.close()

• In the demos above, we just used a hard-coded list of URLs
• In practice, getting to that list of books/products/songs/artists/users (generally speaking: “seeds”) is a web scraper in itself!
• Examples
  • scrape all artists featured on the homepage
  • then loop through them an extract information for each artist (e.g., total number of)
  • scrape user names from reddit.com; then, use the reddit API to get user metadata
  • …

• Many challenges in sampling
  • sample size? generalizability? panel attrition?
  • all of the firm's data? or just a little bit? are subjects vulnerable (say, kids)?
  • can we actually “get” data on all subjects? can we match data?
• Some solutions and best practices discussed in “Fields of Gold”
• Let's look at the tables in the paper and try to find out…

• Have list of seeds to start up data collection (here, artist IDs)
• loop through list of URLs
  • store raw data for diagnostic purposes
  • extract relevant data to JSON file with raw data
• schedule
• infrastructure

• Get content from a website
• Since we don't want “everything”, we need to “find” relevant elements on the site using
  • classes
  • attributes
  • (attribute-value pairs)
• We first build a prototype; then gradually improve it by
  • modularizing code as much as possible (using functions and loops)
  • ensuring code runs top-down

• Standard way for exchanging data, functions or algorithms
• Which APIs did you already encounter/explore?
• Music to scrape is super easy to use - other APIs require advanced authentication procedures
• Structure corresponds to web scraping
  • have list of seeds (e.g., artist IDs; this was “URLs” for web scraping earlier)
  • store data in new-line separated JSON files (if necessary: convert back to CSV)

• Let's view the documentation

• Let's explore a first endpoint: https://api.music-to-scrape.org/artists/featured

–> we can use the browser to retrieve data from (simple) APIs

Extensions:

• time data retrieval (e.g., every five seconds; discuss multiple strategies)
• store featured artists in a list (or file)

• Endpoint: https://api.music-to-scrape.org/artist/info
• Requires parameter: artistid
• Combined call: https://api.music-to-scrape.org/artist/info?artistid={ENTER ARTIST ID HERE}

DO:

• Get some data for an artist of your choice - you find the artist IDs on the site or in previous code
• Wrap code in a function so you can retrieve data for multiple artists

• Here, we extracted data every few seconds.
• But, extraction frequency vastly differs by project
• Considerations
  • archival vs. live data?
  • at which frequency does your phenomenon occur?
  • what's the refresh rate of the data source?
  • any excessive burden on server's caused by frequency of extraction?

• Some solutions
  • explore gains of live data collections
  • adhere to best practices (say, 1 req per second)
  • randomize extraction order
  • use automatic schedulers for consistency

Please work on exercise 2.4.

1. collect (longer) list of featured artists
2. for each of these featured artists, collect meta data
3. store data in new-line separated JSON file

• Processing can have various degrees
  • just “save” the raw data
  • extract only necessary information
  • choose data format for saving (say, JSON vs. CSV)
• Some selected challenges
  • GDPR vs value in retaining raw data
  • Anonymization or pseudonymization required?

• Solutions
  • Retain raw when possible
  • Parse minimal amount of data on the fly
  • Remove sensitive info
  • Ensure proper encoding

• Work on team activity #1
  • coaching session following today's tutorial
  • make team allocation definite (see Canvas!)
• If you haven't done so…
  • go through the self-study material of this week
• Get engaged!
  • discuss ideas, post solutions, identify business opportunities, etc.
  • try fiddling with an AI API or use ChatGPT for starting code
  • be in touch on WhatsApp for any issues/bugs/etc.