DEV Community

Vicente Maldonado
Vicente Maldonado

Posted on • Originally published at Medium on

Beautiful Soup Hello World

Beautiful Soup is a Python library for working with HTML and XML files. You can use it to navigate a HTML document, search it, extract data from it and even change the document structure. Let’s see how it works:

from bs4 import BeautifulSoup

html = '''
    <html>
    <head>
        <title>Beautiful Soup Hello World</title>
    </head>
    <body>
        <h1>Header</h1>
        <p>Paragraph 1</p>
        <p>Paragraph 2</p>
        <p>Paragraph 3</p>
    </body>
    </html>
'''

soup = BeautifulSoup(html, 'html.parser')

print(soup.title)
print(soup.title.name)
print(soup.title.text)

print(soup.p.text)

for paragraph in soup.find\_all('p'):
    print(paragraph.text)

print(soup.get\_text())
Enter fullscreen mode Exit fullscreen mode

It’s a really basic example, but before you can run it you first need to install Beautiful Soup:

pip install beautifulsoup4
Enter fullscreen mode Exit fullscreen mode

While you’re at it, install another library as well:

pip install lxml
Enter fullscreen mode Exit fullscreen mode

It’s a HTML/XML parser. Don’t worry about it.

Let’s start — import Beautiful Soup:

from bs4 import BeautifulSoup
Enter fullscreen mode Exit fullscreen mode

Next, we need some HTML to work with:

html = '''
    <html>
    <head>
        <title>Beautifulsoup Hello World</title>
    </head>
    <body>
        <h1>Header</h1>
        <p>Paragraph 1</p>
        <p>Paragraph 2</p>
        <p>Paragraph 3</p>
    </body>
    </html>
'''
Enter fullscreen mode Exit fullscreen mode

It is a basic HTML document stored in a Python string. Of course, working with HTML stored in a Python script is not very exciting, but this is a Hello, World, so hey.

Create an instance of the BeautifulSoup object, specifying the HTML document and the parser to be used (I said don’t worry about it):

soup = BeautifulSoup(html, 'html.parser')
Enter fullscreen mode Exit fullscreen mode

Now we have our HTML parsed and stored in a variable named soup and we can play with it:

print(soup.title)
Enter fullscreen mode Exit fullscreen mode

Use soup.title to access the HTML document’s

element. This prints:
<title>Beautifulsoup Hello World</title>

Sometimes you don’t want the HTML tag:

print(soup.title.text)

and get just the element text:

Beautifulsoup Hello World

Our document has just one

element so Beautiful Soup appropriately returns it if we use soup.title. But the documents has three

elements (paragraphs) so what happens when we try to pull the same trick?

print(soup.p.text)

It returns the first

element in the document:

Paragraph 1

If you want to get all paragraphs in a documents, well, just use find_all():

for paragraph in soup.find\_all('p'):
    print(paragraph.text)

find_all() returns all paragraphs in the document and you can iterate them using a simple for loop.

This is just scratching the surface with Beautiful Soup. At the end let’s see how simple it is to get all text (and only text) in the document:

print(soup.get\_text())

As expected, this prints

Beautifulsoup Hello World

Header
Paragraph 1
Paragraph 2
Paragraph 3

You can find the full script in my Github. ttfn.

Top comments (0)