Web Scraping with JavaScript

Web Scraping with JavaScript

Web scraping is something I considered back when I was working with Python and found out about BeautifulSoup and Scrapy. When a website doesn't have a way to request/retrieve information programmatically (like with an API), an alternative way of "requesting" the data is by scraping it, or collecting it by using a program or script. There are legal considerations when it comes to web scraping, so I'll start this off by sharing a video about that:

The latest information about the legality of web scraping based on a court decision is "any data that is publicly available and not copyrighted is fair game for web crawlers":

Web scraping is now legal
In late 2019, the US Court of Appeals denied LinkedIn’s request to prevent HiQ, an analytics company, from scraping its data. The decision was a historic moment in the data privacy and data…

Now that you've watched and read that, here's a video that explains what web scraping is:

Here's some more detailed info from Wikipedia.

Okay, so now you know what you're legally able to scrape and what web scraping is. So how do you do it with JavaScript? I've done a bit of searching and put together a list of resources that show you how. Here they are, articles and videos listed from newest (June 2020) to oldest (January 2017):

Web Scraping with Javascript and NodeJS
Learn web scraping with Javascript and NodeJS with this step-by-step tutorial. We will see the different ways to scrape the web in Javascript through lots of example.
Guide to Web Scraping with JavaScript
Web scraping with JavaScript
Building daily food menu app using NextJS, Puppeteer and Zeit Now.
Web scraping for web developers: a concise summary
Knowing one approach to web scraping may solve your problem in the short term, but all methods have their own strengths and weaknesses. Being aware of this can save you time and help you to solve a task more efficiently. Numerous resources exist, which will show you a single technique for extractin…
The Ultimate Guide to Web Scraping with Node.js
So what’s web scraping anyway? It involves automating away the laborious task of collecting information from websites. There are a lot of use cases for web scraping: you might want to collect prices from various e-commerce sites for a price comparison site. Or perhaps you need flight times and hote…
How I built a job scraping web app using Node.js
by Oyetoke Tobi Emmanuel How I built a job scraping web app using Node.js Scraping jobs from the web has now become easier thanks to Indreed. About Indreed Indreed [https://indreed.herokuapp.com] is a Rest API for scraping jobs from Indeed and around the web. It is powered by my personal web scra…
A Guide to Automating & Scraping the Web with JavaScript (Chrome + Puppeteer + Node JS)
In this tutorial you’ll learn how to automate and scrape the web with JavaScript. To do this, we’ll use Puppeteer. Puppeteer is a Node library API that allows us to control headless Chrome. Headless…
Scraping data in 3 minutes with Javascript
Today’s goal will be to scrape some data out of an HTML page and to smartly structure the output data so we can save it right into an hypothetical database. Then we need to organize our data. As you…

Keep in mind that older resources means older info and methods that might not work with today's technologies, but they still offer an insight into the "how to" of web scraping with JavaScript.

Thanks for checking out this post!