DEV Community

Discussion on: Serverless web scraper in Ruby - tutorial

Collapse
 
hguzman profile image
Henry Miguel Guzmán Escorcia • Edited

My code does not work... :(

require 'json'
require 'selenium-webdriver'

def lambda_handler(event:, context:)
  setup_driver
  # driver.navigate.to 'http://www.google.com'
  # element = driver.find_element(name: 'q')
  # element.send_keys 'Pizza'
  # element.submit
  # title = driver.title
  # driver.quit
  { statusCode: 200, body: JSON.generate("Hola mundo") }
end

def setup_driver
    options = Selenium::WebDriver::Chrome::Options.new(binary: 'bin/headless-chromium')
    options.add_argument('--headless')
    options.add_argument('--disable-gpu')
    options.add_argument('--window-size=1280x1696')
    options.add_argument('--disable-application-cache')
    options.add_argument('--disable-infobars')
    options.add_argument('--no-sandbox')
    options.add_argument('--hide-scrollbars')
    options.add_argument('--enable-logging')
    options.add_argument('--log-level=0')
    options.add_argument('--single-process')
    options.add_argument('--ignore-certificate-errors')
    options.add_argument('--homedir=/tmp')
    service = Selenium::WebDriver::Service.chrome(path: 'bin/chromedriver')
    @driver = Selenium::WebDriver.for :chrome, service: service, options: options
    # @driver.manage.timeouts.implicit_wait = 30
end
Enter fullscreen mode Exit fullscreen mode
START RequestId: c6fca6cd-e2e6-4782-8ca6-e10197058471 Version: $LATEST
Error raised from handler method{
  "errorMessage": "unable to connect to chromedriver 127.0.0.1:9515",
  "errorType": "Function<Selenium::WebDriver::Error::WebDriverError>",
  "stackTrace": [
    "/var/task/vendor/bundle/ruby/2.7.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/service.rb:200:in `connect_until_stable'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/service.rb:111:in `block in start'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/socket_lock.rb:41:in `locked'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/service.rb:108:in `start'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:303:in `service_url'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/chrome/driver.rb:40:in `initialize'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:46:in `new'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:46:in `for'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver.rb:88:in `for'",
    "/var/task/prueba.rb:30:in `setup_driver'",
    "/var/task/prueba.rb:5:in `lambda_handler'"
  ]
}END RequestId: c6fca6cd-e2e6-4782-8ca6-e10197058471
REPORT RequestId: c6fca6cd-e2e6-4782-8ca6-e10197058471  Duration: 20236.86 ms   Billed Duration: 20300 ms   Memory Size: 512 MB Max Memory Used: 62 MB  Init Duration: 303.13 ms    
Enter fullscreen mode Exit fullscreen mode
source 'https://rubygems.org'
gem 'selenium-webdriver'
Enter fullscreen mode Exit fullscreen mode

Chrome81.0.4044.138
stable-headless-chromium-amazonlinux-2017-03.zip

Lambda AWS

Please help me.

Collapse
 
mknycha profile image
Marcin K.

Hi,

Sorry for the late reply.
Your code and Gemfile are ok.
It looks like you're running it on Ruby 2.7 in lambda and it's not compatible with this chromedriver version.
Unfortunately, chromedriver must be compatible with your serverless chrome and ruby version, it's not easy to find a match.
The easiest solution, for now, would be to downgrade to ruby 2.5 in lambda - just create a new lambda function with this version.

Collapse
 
chaebyunghoon profile image
Hoonki

Hello. from now on, aws plan to deprecate ruby 2.5.
So we have to migrate version of ruby 2.5 to 2.7.
How can i find compatible chromedriver version with ruby 2.7? Can you send me an reference? Thank you.