What is Web Scraping FYI?

Updated

Web Scraping FYI is a web scraping programming knowledge base that condenses popular scraping concepts, tools, and techniques into a single navigable site. It is aimed at both beginner and experienced developers who want an organized reference rather than scattered tutorials. The site is informational only — it does not appear to sell a product or require an account based on the available page content.

What the site covers

The knowledge base is structured into several sections, each addressing a different layer of scraping work:

  • Concepts — What web scraping is, how it differs from web crawling, and whether it is legal.
  • How to scrape — Scraping static pages, parsing HTML data, finding hidden data, scraping dynamic pages, automating browsers, avoiding blocks, and scaling a scraper.
  • Tools & technologies — Languages and HTTP clients, scraping frameworks, browser automation, browser libraries, anti-bot protections, scraping APIs, and developer tools.
  • How the web works — HTTP, HTML, JavaScript, and JSON fundamentals.
  • Libraries — Organized by language, including Python, JavaScript, PHP, Go, Ruby, and R.
  • Site-specific guides — Step-by-step scraping instructions grouped by category.

Site-specific scraping guides

A large part of the content is organized around individual websites, which is useful if you already know your target and want a starting point.

Category Examples of covered sites
E-commerce Amazon, Walmart, eBay, Etsy, AliExpress, Best Buy, StockX, Nordstrom
Real estate Zillow, Realtor.com, Redfin, Zoopla, Rightmove, Idealista
Social media Instagram, TikTok, Twitter/X, Reddit, Threads, YouTube
Jobs & business LinkedIn, Indeed, Glassdoor, Wellfound, ZoomInfo, Crunchbase
Reviews & travel Trustpilot, Yelp, YellowPages, TripAdvisor, Booking.com
Search & other Google, Bing, SimilarWeb, Domain.com

Who it is for

The site states it is designed for both beginner and experienced developers. In practice:

  • Beginners can use the concepts and "how the web works" sections to build background before writing a scraper.
  • Experienced developers can jump to the tools, libraries, or site-specific guides for a quick reference on a particular target or technique.

How to use it

  1. Start with the concepts section if you are new to scraping, or skip straight to the how-to section if you already understand the basics.
  2. Pick the guide matching your target site category to see the specific approach for that site.
  3. Cross-reference the tools and libraries sections when choosing a language, HTTP client, or automation framework.

The main limitation to keep in mind is that the page evidence describes this as an overview and knowledge base, so it is best treated as a starting point and map of the field rather than a substitute for testing against a live site.