oDCM - Web Scraping 101 (Tutorial)

Extend the code snippet below to extract the release year of the song below.

import requests
from bs4 import BeautifulSoup
url = 'https://music-to-scrape.org/song?song-id=SOJZTXJ12AB01845FB'
request = requests.get(url, headers =  {'User-agent': 'Mozilla/5.0'})
song_page = BeautifulSoup(request.text)
about = song_page.find(class_='about_artist')
title = about.find('p')

• Use the browser's inspect mode to find the relevant attributes
• Consecutively develop the code step-by-step, “onion”-like

about.find_all('p')

[<p>China</p>, <p>Tito Puente</p>, <p>1972</p>, <p>193</p>]

plays=about.find_all('p')[3].get_text()

Recommendations:

• Practice even more, e.g., with new information that you haven't extracted before from this and other sections of the website.
• Always build code gradually, never from scratch

• Any issues you've run into?
• Solutions posted on the site!

For today's tutorial, we will be sampling users from the main website of music-to-scrape.org

Can you propose a strategy to capture them? Any attributes/classes you see?

Please use the code snippet below to extract the 'href' attribute of all links on the website.

• Links are identifiable by the “a” tag.
• The relevant attribute is called “href”
• Build your code consecutively!

url = 'https://music-to-scrape.org'

res = requests.get(url)
res.encoding = res.apparent_encoding

homepage = BeautifulSoup(res.text)

# continue here

users = []
for link in relevant_section.find_all("a"):
  if ('href' in link.attrs):
      users.append(link.attrs['href'])
users

['user?username=Shadow49', 'user?username=Star47', 'user?username=Coder43', 'user?username=SonicShadow61', 'user?username=CoderGeek94', 'user?username=StealthWizard02']

• Let's now take a look at these links more closely.

• Notice that the links were relative links and we won't be able to use them for looping (e.g., try pasting them in your browser - they won't work!)
• So, let's turn them into absolute links, simply by concatenating (= combining) strings.

urls = []
for link in relevant_section.find_all("a"):
    if 'href' in link.attrs: 
        extracted_link = link.attrs['href']
        urls.append(f'https://music-to-scrape.org/{extracted_link}')
urls

['https://music-to-scrape.org/user?username=Shadow49', 'https://music-to-scrape.org/user?username=Star47', 'https://music-to-scrape.org/user?username=Coder43', 'https://music-to-scrape.org/user?username=SonicShadow61', 'https://music-to-scrape.org/user?username=CoderGeek94', 'https://music-to-scrape.org/user?username=StealthWizard02']

import requests
from bs4 import BeautifulSoup

def get_users():
    url = 'https://music-to-scrape.org/'

    res = requests.get(url)
    res.encoding = res.apparent_encoding

    soup = BeautifulSoup(res.text)

    relevant_section = soup.find('section',attrs={'name':'recent_users'})

    links = []
    for link in relevant_section.find_all("a"):
        if 'href' in link.attrs: 
            extracted_link = link.attrs['href']
            links.append(f'https://music-to-scrape.org/{extracted_link}')
    return(links) # to return all links

# let's try it out
# users = get_users() 

• Functions allow us to re-use code; it prevents errors, and helps us structure our code.

import json

users = get_users()
# build JSON dictionary
f = open('users.json','w')
for user in users:
  obj = {'url': user,
         'username': user.replace('https://music-to-scrape.org/user?username=','')}
  f.write(json.dumps(obj))
  f.write('\n')

85
1
81
1
83
1
95
1
91
1
99
1

f.close()

links = []
for link in relevant_section.find_all("a"):
    if 'href' in link.attrs: 
        extracted_link = link.attrs['href']
        links.append(f'https://music-to-scrape.org/{extracted_link}')

# getting songs
songs = []
for song in relevant_section.find_all("span"):
    songs.append(song.get_text())

len(links)

6

len(songs)

4

How can you capture the song data, such that we can always link the particular song to particular users? (i.e., to prevent array misalignment)?

Tip: Observe how we can first iterate through users, THEN extract links (similar to how we have done it with relevant_section).

users = relevant_section.find_all(class_='mobile-user-margin')
data = []
for user in users: # iterate through each user
  obj = {'link':user.find('a').attrs['href']  }
  data.append(obj)
data

[{'link': 'user?username=Shadow49'}, {'link': 'user?username=Star47'}, {'link': 'user?username=Coder43'}, {'link': 'user?username=SonicShadow61'}, {'link': 'user?username=CoderGeek94'}, {'link': 'user?username=StealthWizard02'}]

users = relevant_section.find_all(class_='mobile-user-margin')
data = []
for user in users:
  if user.find('span') is not None:
    song_name=user.find('span').get_text()
  else:
    song_name='NA'
  obj = {'link':user.find('a').attrs['href'],
         'song_name': song_name}

  data.append(obj)
data

[{'link': 'user?username=Shadow49', 'song_name': 'John Stewart - Price Of The Fire'}, {'link': 'user?username=Star47', 'song_name': 'NA'}, {'link': 'user?username=Coder43', 'song_name': 'Francis Dunnery - Too Much Saturn'}, {'link': 'user?username=SonicShadow61', 'song_name': 'NA'}, {'link': 'user?username=CoderGeek94', 'song_name': 'Lonnie Johnson - Raining On The Cold_ Cold Ground'}, {'link': 'user?username=StealthWizard02', 'song_name': 'Pitch Black - Harmonia'}]

• Two strategies
  • Understand how links are built (pre-building them; strategy 1)
  • Understand how to “click” to the next/previous page (consecutively building them; strategy 2)

urls = []
counter = 37
while counter >= 0:
  urls.append(f'https://music-to-scrape.org/user?username=StarCoder49&week={counter}')
  counter = counter - 1
urls

['https://music-to-scrape.org/user?username=StarCoder49&week=37', 'https://music-to-scrape.org/user?username=StarCoder49&week=36', 'https://music-to-scrape.org/user?username=StarCoder49&week=35', 'https://music-to-scrape.org/user?username=StarCoder49&week=34', 'https://music-to-scrape.org/user?username=StarCoder49&week=33', 'https://music-to-scrape.org/user?username=StarCoder49&week=32', 'https://music-to-scrape.org/user?username=StarCoder49&week=31', 'https://music-to-scrape.org/user?username=StarCoder49&week=30', 'https://music-to-scrape.org/user?username=StarCoder49&week=29', 'https://music-to-scrape.org/user?username=StarCoder49&week=28', 'https://music-to-scrape.org/user?username=StarCoder49&week=27', 'https://music-to-scrape.org/user?username=StarCoder49&week=26', 'https://music-to-scrape.org/user?username=StarCoder49&week=25', 'https://music-to-scrape.org/user?username=StarCoder49&week=24', 'https://music-to-scrape.org/user?username=StarCoder49&week=23', 'https://music-to-scrape.org/user?username=StarCoder49&week=22', 'https://music-to-scrape.org/user?username=StarCoder49&week=21', 'https://music-to-scrape.org/user?username=StarCoder49&week=20', 'https://music-to-scrape.org/user?username=StarCoder49&week=19', 'https://music-to-scrape.org/user?username=StarCoder49&week=18', 'https://music-to-scrape.org/user?username=StarCoder49&week=17', 'https://music-to-scrape.org/user?username=StarCoder49&week=16', 'https://music-to-scrape.org/user?username=StarCoder49&week=15', 'https://music-to-scrape.org/user?username=StarCoder49&week=14', 'https://music-to-scrape.org/user?username=StarCoder49&week=13', 'https://music-to-scrape.org/user?username=StarCoder49&week=12', 'https://music-to-scrape.org/user?username=StarCoder49&week=11', 'https://music-to-scrape.org/user?username=StarCoder49&week=10', 'https://music-to-scrape.org/user?username=StarCoder49&week=9', 'https://music-to-scrape.org/user?username=StarCoder49&week=8', 'https://music-to-scrape.org/user?username=StarCoder49&week=7', 'https://music-to-scrape.org/user?username=StarCoder49&week=6', 'https://music-to-scrape.org/user?username=StarCoder49&week=5', 'https://music-to-scrape.org/user?username=StarCoder49&week=4', 'https://music-to-scrape.org/user?username=StarCoder49&week=3', 'https://music-to-scrape.org/user?username=StarCoder49&week=2', 'https://music-to-scrape.org/user?username=StarCoder49&week=1', 'https://music-to-scrape.org/user?username=StarCoder49&week=0']

Run the code below. How to extend the code to extract the LINK of the previous button?

url = 'https://music-to-scrape.org/user?username=StarCoder49&week=36'
header = {'User-agent': 'Mozilla/5.0'}
res = requests.get(url, headers = header)
res.encoding = res.apparent_encoding
userpage = BeautifulSoup(res.text)
button=userpage.find(class_='page-link', attrs={'type':'previous_page'})

• We now have a function get_users() to retrieve user names
• We also know how to “find” the next link to visit (a user's previous page)
• How can we now visit ALL pages, for ALL users?
• Let's take a look at some pseudo code!

users = get_users()

consumption_data = []

for user in users:
  url = user['url']

  while url is not None:
    # scrape information from URL # challenge #2.1, #2.4
    # determine "previous page"
    time.sleep(1) # challenge #2.3

    # if previous page exists: rerun loop on next URL
    # if previous page does not exist: stop while loop, go to next 

• So far, we have done this

  • requests (to get) –> beautifulsoup (to extract information, “parse”)

• DO: Run the snippet below and open amazon.html

import requests
header = 
f = open('amazon.html', 'w', encoding = 'utf-8')
f.write(requests.get('https://amazon.com', headers =  {'User-agent': 'Mozilla/5.0'}).text)
f.close()

Can you explain what happened?

• Many dynamic sites require what I call “simulated browsing”

• Try this:

!pip install webdriver_manager
!pip install selenium

# Using selenium 4 - ensure you have Chrome installed!
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))

url = "https://amazon.com/" # remember to solve captchas
driver.get(url)

• See your browser opening up?
• Beware of rerunning code - a new instance will run each time!