TLDR;
A Nokogiri is a Japanese pull saw. Oh... didn't help? Keep reading.
The Gem
Nokogiri comes standard in the Gemfiles of Rails apps. I first noticed it because it showed up in a Bundler error.
Nokogiri is a low-dependency module written in C, Java, and Ruby that parses XML and HTML so that these protocols can be used by Ruby. Let's get started.
Using Nokogiri to Parse HTML
Do you want to use Ruby to extract data from a webpage?
First run nokogiri -v and see if you get a response with a verion number. If not, you'll have to run gem i nokogiri
➜ ~ irb --simple-prompt
>> require 'nokogiri'
=> true
>> Nokogiri
=> Nokogiri
Don't forget to require it, otherwise you'll get a NameError for an uninitialized constant.
First, we will start by parsing HTML we create, then we will parse an actual webpage.
# First, put some valid html into a string.
>> html = "<h1>Hello World!</h1>
<section><h2>A List of Stuff</h2>
<ul><li>Camera</li>
<li>Computer</li>
<li>Television</li>
</ul></section>"
# Now pass that string into the HTML5 module as an argument
>> document = Nokogiri::HTML5(html)
You should get a response like this
=>
#(Document:0x4c4 {
name = "document",
children = [
#(Element:0x4d8 {
name = "html",
children = [
#(Element:0x4ec { name = "head" }),
#(Element:0x500 {
name = "body",
children = [
#(Element:0x514 {
name = "h1",
children = [ #(Text "Hello World!")]
}),
#(Element:0x528 {
name = "section",
children = [
#(Element:0x53c {
name = "h2",
children = [ #(Text "A List of Stuff")]
}),
#(Element:0x550 {
name = "ul",
children = [
#(Element:0x564 {
name = "li",
children = [ #(Text "Camera")]
}),
#(Element:0x578 {
name = "li",
children = [ #(Text "Computer")]
}),
#(Element:0x58c {
name = "li",
children = [ #(Text "Television")]
})]
})]
})]
})]
})]
})
Nokogiri's HTML5 module took our HTML string and returned a Document class to us that has it's own set of methods. Let's look at a couple. (Remember you can always list all the methods by using Document.methods.sort.)
Document#children
This will will return an array-like XML::NodeSet. It only has a length of one because it's the <html>...</html> element that we never included in our string. This child has two children, the <head>...</head> and the <body>...</body> tags.
>> body = document.children.children.last
=>
#(Element:0x80c {
...
>> ul = body.children.children.children
=> [#<Nokogiri::XML::Text:0x8c0 "A List of Stuff">, #<Nokogiri::XML::Elemen...
?> => [#<Nokogiri::XML::Text:0x8c0 "A List of Stuff">, #<Nokogiri::XML::Elemen..
.
>> >> ul.children.length
>> => 3
?> ?> ul.children.each do |child|
>> ?> puts child.text
>> >> end
>> Camera
>> Computer
>> Television
Parsing HTML from the Internet
I've heard the Internet is a good place to find HTML documents to parse. You can capture some of this HTML by using the Client URL command in your terminal.
Ruby-Doc.org is a great resource to learn about all Ruby objects.
Capture it like this.
➜ ~ curl 'https://ruby-doc.org'
While it's cool to see the HTML populate the terminal, it's even better when it comes back as a Ruby object for us to use the data. To do so, let's use Nokogiri!
# You'll need the open-uri gem in addition to Nokogiri
# Open-uri allows you to open a URI as if it was a file in Ruby
>> require 'nokogiri'
=> true
>> require 'open-uri'
=> true
>> doc = Nokogiri::HTML5(URI.open('https://ruby-doc.org'))
=>
#(Document:0x9998 {
...
>>
Now we can use the #css method to query the html by elements and classes.
>> puts doc.css('h1')
<h1><a href="/">Ruby-Doc.org</a></h1>
=> nil
>> puts doc.css('p').first
<p>Help and documentation for the Ruby programming language.</p>
Query class this way.
>> rails = Nokogiri::HTML5(URI('https://rubyonrails.org'))
=>
#(Document:0x1f1bc {
...
>> headline = rails.css('div.heading__headline')
=> [#<Nokogiri::XML::Element:0x1fb94 name="div" attributes=[#<Nokogiri::XML...
>> puts headline.text
Compress the complexity of modern web apps.
Learn just what you need to get started, then keep leveling up as you go. Ruby on Rails scales from HELLO WORLD to IPO.
You’re in good company.
Over the past two decades, Rails has taken countless companies to millions of users and billions in market valuations.
Building it together.
Over six thousand people have contributed code to Rails, and many more have served the community through evangelism, documentation, and bug reports. Join us!
Everything you need.
Rails is a full-stack framework. It ships with all the tools needed to build amazing web apps on both the front and back end.
Optimized for happiness.
Let’s get started.
Great! Now you have a general idea of how Nokogiri works!
And now for free bonus content...
Let's search a webpage's content. For instance, it's basketball season at the time of this writing, so let's see if the homepage to ESPN talks about basketball.
>> espn = Nokogiri::HTML5(URI('https://www.espn.com'))
=>
#(Document:0x259e0 {
...
# For this we will use a Regular Expression (RegEx)
# The Regular Expression goes between forward slashes
# And the 'i' flag makes the search case-insensitive
>> espn.text.scan(/basketball/i).count
=> 32
There you have it! And we didn't even have to visit the website to find out that ESPN refers to basketball 32 times on it's homepage! Wow...
Top comments (0)