Beautiful Soup
Beautiful Soup is a library that makes it easy to scrape information from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree. Beautiful Soup's support for Python 2 was discontinued on December 31, 2020: one year after the sunset date for Python 2 itself. From this point onward, new Beautiful Soup development will exclusively target Python 3. The final release of Beautiful Soup 4 to support Python 2 was 4.9.3. Beautiful Soup is licensed under the MIT license, so you can also download the tarball, drop the bs4/ directory into almost any Python application (or into your library path) and start using it immediately.
Learn more about Beautiful Soup
Jaunt
Jaunt is a Java library designed for web scraping, web automation, and JSON querying. It provides a fast, ultra-light headless browser that enables Java programs to perform tasks such as web scraping, form handling, and interfacing with REST APIs. Jaunt supports parsing of HTML, XHTML, XML, and JSON, and offers features like HTTP header and cookie manipulation, proxy support, and customizable caching. The library does not support JavaScript execution; however, for automating JavaScript-enabled browsers, Jauntium is recommended. Jaunt is available under the Apache License, with a monthly edition that expires periodically, requiring users to download the latest version upon expiration. The library is suitable for tasks such as parsing and extracting data from web pages, filling out and submitting forms, and handling HTTP requests and responses. Comprehensive tutorials and documentation are available to assist users in getting started with Jaunt.
Learn more about Jaunt
WebScraping.AI
WebScraping.AI is an AI-powered web scraping API that simplifies data extraction by handling browsers, proxies, CAPTCHAs, and HTML parsing on behalf of the user. By providing a URL, users can receive the HTML, text, or data from the target webpage. The platform features JavaScript rendering in a real browser, ensuring that page content appears exactly as it would on a user's computer. It also offers automatically rotated proxies, allowing users to scrape any site without limitations, with geotargeting options available. HTML parsing is performed on WebScraping.AI's servers, alleviating concerns about heavy CPU load and potential vulnerabilities in HTML parsers. Additionally, the platform includes tools powered by large language models to extract unstructured page content, provide answers to questions, generate summaries, and perform rewrites. Users can extract visible page text after JavaScript rendering and use it as a prompt for their own LLM models.
Learn more about WebScraping.AI
parsel
Parsel is a BSD-licensed Python library to extract and remove data from HTML and XML using XPath and CSS selectors, optionally combined with regular expressions. Create a selector object for the HTML or XML text that you want to parse. Then use CSS or XPath expressions to select elements. CSS is a language for applying styles to HTML documents. It defines selectors to associate those styles with specific HTML elements. XPath is a language for selecting nodes in XML documents, which can also be used with HTML. You can use either CSS or XPath. CSS is usually more readable, but some things can only be done with XPath. Being built atop lxml, parsel selectors support some EXSLT extensions and come with pre-registered namespaces to use in XPath expressions. Parsel selectors allow you to chain selectors, so most of the time you can just select by class using CSS and then switch to XPath when needed.
Learn more about parsel