oDCM - Opening Lecture

Using virtual environment '/Users/hannesdatta/.virtualenvs/r-reticulate' ...

Using virtual environment '/Users/hannesdatta/.virtualenvs/r-reticulate' ...

We're about to start with the first lecture of this class.

If you haven't done so, please

• explore the course page at https://odcm.hannesdatta.com.
• complete the software installation (https://odcm.hannesdatta.com/docs/modules/prep/)
• my slides are available here: https://odcm.hannesdatta.com/docs/modules/week1

• Part 1 (12.45 to about 13.45)
  • Getting to know each other
  • Motivation for the course
  • Course framework and learning goals
  • Agenda and practical arrangements
• Break
• Part 2: Python Bootcamp on your laptops (about 14.00 - 15.30/15.45)

• This is not a class that merely teaches you Python (but, if you invest time, you'll learn Python on the way!)
• You can also extract web data using other software packages (e.g., R)
• Mix of students at various levels (e.g., beginners, advanced Python users)
• Course mostly takes place on campus (based on student feedback); attendance is not mandatory but strongly encouraged
• I will not record any online or offline sessions; slides will be posted always
• Consider me your coach, not your distant professor
• Slow me down if you need to

• scraping nerd — learned it in 2008 using Visual Basic in Excel
• started doing my own research with scraped and API-extracted data in 2012 (so, 10+ years experience)
• left Germany around your age, now many years in NL
• Associate Professor at Tilburg University

• Substantive interests

  • streaming business models (e.g., music, movies)
  • marketing-mix modeling and optimization
  • open science

• Methodological interests

  • online data collection via APIs and web scraping
  • causal effects with observational data

• MSc Marketing Analytics
  • Data preparation and workflow management (https://dprep.hannesdatta.com)
  • Online data collection and management (https://odcm.hannesdatta.com)
  • MSc Thesis supervision (https://thesis.hannesdatta.com)
• Other initiatives
  • Tilburg Science Hub (https://tilburgsciencehub.com)
  • Tilburg.ai (https://tilburg.ai)
  • Music-to-scrape.org (https://music-to-scrape.org)
  • YouTube channel (https://youtube.com/c/hannesdatta)
  • My code (https://github.com/hannesdatta)
  • More about my work (https://hannesdatta.com)

• What's your background - previous education (e.g., program)?
• Any experience in Python (or other programming languages)?
• What are your passions & talents? (+ why I am asking you this…)

• started out as a PhD student without data
• was interested in music, and found website with data (https://last.fm)
• no best practices in scraping; learnt all by myself and made many mistakes
• scraping undervalued in academic job market, but, key role in shaping relevance and rigor of your work
• now scraping and APIs are a large part of what defines me…

• investigate music streaming (How did music consumption change with Spotify?)
• monitored Spotify's user interface and make lists of new releases
• investigate playlist ecosystem with APIs
• scraped reviews at Amazon.com
• investigating “video streaming wars” (e.g., Netflix versus Disney+, when do consumers use multiple streaming services?), see trakt.tv
• share code on GitHub
• developed a methodological framework on scraping/APIs (which we use in this class)
• Faced legal battles…

With web scraping, you can capture anything you can view in a web browser

• pricing data at bol.com
• reviews at Amazon.com
• movie data at imdb.com or reviews at Rotten Tomatoes

With APIs, you obtain official data from a firm in a programmatic way

• e.g., as a developer, interact with Instagram, Twitter/X, ChatGPT / OpenAI, AWS, …
• as a researcher, construct data set from analytics firms

• for businesses
  • “stitch together” different services (e.g., augment functionality of ChatGPT)
  • do market research (e.g., pricing data; see Zyte)
  • initialize recommendation systems (e.g., music metrics)
• for research
  • discover/document novel phenomena (e.g., new platforms/technologies)
  • improving methods (e.g., get data to try out new methods, such as review data & text analysis, images, videos)
  • improved inferences (i.e., get more accurate results)
  • collect metrics managers care about

• What are cool websites/services you're using often?!

• What are important societal issues right now that directly or indirectly affect your lives?

• As a marketer, how could you use the API of OpenAI to automate/invent something new?

Let's talk about it right now…

Why do we need a course on this? Isn't this how research is always done?

Yes, but collecting web data is different from other datasets!

• data source selection
  • finding the right data source may be difficult as many potential alternatives exist
  • differ in delivery formats (website versus API, compared to CSV/databases)
  • access to data that is not available commercially (or that a firm would not like to share)

• extraction design
  • which information to select (which is available?!)
  • type of variables (sales is rare; review scores are abundant)
  • different stakeholders that could potentially be addressed
  • need to tackle legal and ethical issues

• collecting data at scale!
  • unprecedent: it's totally automatic (but prone to errors)
  • need to put monitoring procedures in place
  • data is not documented, and there is no direct way to ask questions about the data (sampling?! generalizability?)

Each project is totally unique - that's why there is no universal “best way” to approach things…

• Explain how to use web data for creating marketing insight
• Select web data sources and evaluate their value to inform a specific research context or business problem
• Design the web data collection while balancing validity, technical feasibility, and exposure to legal/ethical risks
• Collect data via web scraping and Application Programming Interfaces (APIs) by mixing, extending, and repurposing code snippets
• Document and archive collected data and make it available for public (re)use

• Weekly modules, structured along the methodological framework

  • On-campus lectures and tutorials
  • Self-study
  • On-campus (sometimes online) coaching sessions

• Project in which you put into practice your skills (50% of your grade)

• On-campus computer exam (50% of your grade)

• Bonus points available (max .5 on final course grade; contribute to course or Tilburg Science Hub using source code: e.g., solutions for assignments, writing a tutorial for a new package, debugging source code, maintaining course's issue board); needs GitHub for this

• We're currently experiencing many new retailers that are invented/launched.
• You know way more than I do… so… let's hear some!

Goals:

In the context of retail-based platforms,

• understand and present “data universe”
• select one or multiple data source(s) and collect the data (scraping, APIs)
• document the resulting data for public (re)use

Find your sweet spot!

• Read Guyt et al. 2024 and discover exiting data sources (or add your own!)
• brainstorm about it…

• Specifics

  • scrape data via web sites or use APIs
  • work through the entire method framework
  • receive feedback by students and me (your coach)
  • inspiration for the setup: past projects
  • legacy project ideas

• Evaluation

  • grading rubric available
  • self- and peer assessment used

Visit https://odcm.hannesdatta.com!

• It's open education, so spread the word!

• Course website is your #1 resource, Canvas only used for

  • posting important announcements,
  • sign up for teams, and
  • submitting data challenges/projects.

• Do all students have access to Canvas?

• Take time to become acquainted with the course (e.g., not using Canvas)
• Can be tough at first, but you will gain experience rapidly!
• It can be overwhelming to follow dPrep and oDCM simultaneously, but the skills you learn are important for your course work, Master Thesis & future career!
• Usually not possible to take these courses at a later moment - do get in touch with program coordinator for questions about course enrollment
• Easing your start-up pain
  • Start preparing early on (the first weeks will be the most challenging!)
  • Have the same group members across dPrep and oDCM
  • Collaborate with each other and try to help one another, also across teams!
  • Be in touch via WhatsApp for Business (add +31 13 466 8938)

• Get you up to speed with both web scraping and APIs (and know when to pick which one!)
• Teach you a bit of Python, but, becoming an expert requires years of practice
• Discuss own work and ideas, but this requires interaction, talking to me, working hard
• Use of open software (usable right away, no admin rights required)
• Bring in your own ideas, don't be afraid to study topics off the main stream!
• I value diversity! Be who you want to be. Say what you want to say.

• Coding can be extremely frustrating if you're starting out
• I tend to become semi-“brain-dead” after a day of coding
• Take breaks! Stop coding. Go for a run. Start again.
• You will learn from your mistakes
• Use ChatGPT, cheat sheets and our support section

→ quick feedback loops in first few weeks

When you run into trouble, this is your way out!

1. Try to find the info on the course website
2. Ask Chat GPT / Google / Stackoverflow
3. Ask friend/classmate (form learning groups)
4. Can it wait? Defer to lecturer or feedback sessions.
5. If it can't wait: be in touch with me (–> next slide!)

• Investment in research skills

  • e.g., collect own data & shape its creation to create relevant and rigorous research

• Essential skills for entrepreneurs

  • e.g., have your own company on the basis of web data and APIs
  • be the linking pin (interface between marketing and analytics)

• Showcast expertise in coding

Please tell me what would make this course a success for you

• We will continue with our Python bootcamp after a break
• Haven't followed the installation guide? Check it out on the course page now! (modules –> preparation)

Any questions so far?!