I had previously written about scraping some stats from basketball reference and when I went back to do it again my links didn't work, meaning I would have to redo some of the work. I didn't particularly love my last exploration with Nokogiri and scraping so I took this as an opportunity to learn and expand my knowledge.
I'm not going to write this as a walkthru since this blog will really walk you thru it and really just posting this to show different ways you can get the information you need.
I feel like finding elements using CSS leads to a more stable approach vs hardcoding the XPath. As long as the class name or id is the same, it won't matter if the site ends up nesting the element and breaking your code.
One thing that I spent a lot of time trying to troubleshoot was my variables were coming up empty despite me doing it all seemingly correct. In the end I realized that if the site hadn't finished loading. I'm assuming there's a safer way to do this vs just using sleep 2 or something.
My Code
require 'kimurai'
require 'json'
class TequilaScraper < Kimurai::Base
@name = 'tqdb_scrap'
@start_urls = ['https://tequilamatchmaker.com/tequilas/2325-fortaleza-blanco']
@engine = :selenium_chrome
@@tequilas = []
def scrape_page
sleep 2
doc = browser.current_response
tequila = doc.css('div.product-actions')
teq_name = tequila.css('h1[itemprop="name"]').text.gsub(/\n/, "")
teq_type = tequila.css('div.product-type a').text.gsub(/\n/, "")
teq_rating_p = tequila.css('ul.product-list__item__ratings li')[0].text.gsub(/\D/, '').gsub(/\n/, "")
teq_rating_c = tequila.css('ul.product-list__item__ratings li')[1].text.gsub(/\D/, '').gsub(/\n/, "")
teq_price_check = tequila.css('div.commerce-price-container div span')[1]
if teq_price_check
teq_price = tequila.css('div.commerce-price-container div span')[1].text.gsub(/\n/, "")
else
teq_price = 'n/a'
end
doc_mid = doc.css('div.container')
teq_image = doc_mid.css('img.product-image').attr('src')
teq_nom = doc_mid.css('div.production-details_product table tbody tr')[0].css('td a').text.gsub(/\n/, "")
doc_mid.search('span.sr-only').each do |spans|
# remove search result spans since it's just a comma
spans.remove
end
teq_agave = doc_mid.css('div.production-details_product table tbody tr')[1].css('td').text.gsub(/\n/, "")
teq_agave_region = doc_mid.css('div.production-details_product table tbody tr')[2].css('td').text.gsub(/\n/, "")
teq_region = doc_mid.css('div.production-details_product table tbody tr')[3].css('td').text.gsub(/\n/, "")
teq_cooking = doc_mid.css('div.production-details_product table tbody tr')[4].css('td').text.gsub(/\n/, "")
teq_extraction = doc_mid.css('div.production-details_product table tbody tr')[5].css('td').text.gsub(/\n/, "")
teq_water = doc_mid.css('div.production-details_product table tbody tr')[6].css('td').text.gsub(/\n/, "")
teq_fermentation = doc_mid.css('div.production-details_product table tbody tr')[7].css('td').text.gsub(/\n/, "")
teq_distillation = doc_mid.css('div.production-details_product table tbody tr')[8].css('td').text.gsub(/\n/, "")
teq_still = doc_mid.css('div.production-details_product table tbody tr')[9].css('td').text.gsub(/\n/, "")
teq_aging = doc_mid.css('div.production-details_product table tbody tr')[10].css('td').text.gsub(/\n/, "")
teq_abv = doc_mid.css('div.production-details_product table tbody tr')[11].css('td').text.gsub(/\n/, "")
teq_other = doc_mid.css('div.production-details_product table tbody tr')[12].css('td').text.gsub(/\n/, "")
tequila = {name: teq_name, type: teq_type, rating_p: teq_rating_p, rating_c: teq_rating_c,
price: teq_price, image_url: teq_image, nom: teq_nom, agave: teq_agave, agave_region: teq_agave_region,
region: teq_region, cooking: teq_cooking, extraction: teq_extraction, water: teq_water, fermentation: teq_fermentation,
distillation: teq_distillation, still: teq_still, aging: teq_aging, abv: teq_abv, other: teq_other}
@@tequilas << tequila #if !@@tequilas.include?(tequila)
end
def parse(response, url:, data: {})
scrape_page
File.open("tequila.json","w") do |f|
f.write(JSON.pretty_generate(@@tequilas))
end
@@tequilas
end
end
TequilaScraper.crawl!
puts 'done scraping'
Initially I was going to get mock data from a React site but opted for this tequila database just to test easily. Using Selenium is a bit overkill for this, but all in all it worked really well. I think next time I'm going to have to try to dig into some Python or JS scraping.
Top comments (0)