DEV Community

Cover image for Web Scraping with Langchain and html2text
Ranjan Dailata
Ranjan Dailata

Posted on

Web Scraping with Langchain and html2text


In this blog post, you will be guided on how to perform a simple web scrapping using the available open source python packages. We are going to make use of the langchain and html2text.


First, we need to make sure to install the langchain and html2text packages.

!pip install -q langchain playwright beautifulsoup4 html2text
Enter fullscreen mode Exit fullscreen mode

Here's the code snippet for accomplishing the web scrapping. The following code is utilizing the langchain's AsyncHtmlLoader and the Html2TextTransformer from html2text package for the extraction of HTML to Text.

import html2text
from langchain.document_loaders import AsyncHtmlLoader
from langchain.document_transformers import Html2TextTransformer

async def do_webscraping(link):
        urls = [link]
        loader = AsyncHtmlLoader(urls)
        docs = loader.load()

        html2text_transformer = Html2TextTransformer()
        docs_transformed = html2text_transformer.transform_documents(docs)

        if docs_transformed != None and len(docs_transformed) > 0:
            metadata = docs_transformed[0].metadata
            title = metadata.get('title', '')
            return {
                'summary': docs_transformed[0].page_content,
                'title': title,
                'metadata': metadata,
                'clean_content': html2text.html2text(docs_transformed[0].page_content)
            return None

    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        return None
Enter fullscreen mode Exit fullscreen mode

Let's see the real code usages now. Assume below are the URLs for which you wish to do the web scrapping.

google_search_results = ['',
Enter fullscreen mode Exit fullscreen mode

Here's the logic for interacting and performing the web scrapping.

for link in google_search_results:
  response = await do_webscraping(link)
  if response != None:
Enter fullscreen mode Exit fullscreen mode


Here's the structured response.


Top comments (3)

hilmanski profile image

Interesting! I didn't know about html2text before, thanks for the reference.

I did a similar experiment a while ago, but using AI model from OpenAI to parse the data with structure format here: Web parsing with AI

Thank you for sharing.

ranjancse profile image
Ranjan Dailata

@hilmanski thanks a lot for your feedback. I have published a blog post which mimics the same using LLM -

hilmanski profile image

Thank you Ranjan! I'll read the post!