DEV Community

loading...

Automated Dapps Scrapping with Selenium and Metamask

Leonardo Teixeira Menezes
Passionate Software Engineer
・3 min read

One of the newest trends in web development is the rise of decentralized applications, also known as Dapps. These applications are built leveraging decentralized networks in order to provide trustless interactions between users, using predefined interactions built as smart contracts (if you want to know more about dapps head over here).

To access Dapps users need to use a crypto wallet to connect with, this creates a new challenge for developers who want to scrap and/or test Dapps using tools like Selenium. In this post we will cover the basics on how to solve this using Python and Chromium, however, the principles described here can be applied using any programming language and web browser automation tool.

Most current Dapps depend on a crypto wallet being present in the user browser as an extension, injecting in the web page information about the user wallet and the network it's connected with. One of the most popular browser crypto wallet is Metamask. In order to successfully scrape a Dapp we need to interact not only with the target website but also with the Metamask extension simultaneously, to approve the app connection with our wallet and other possible transactions.

Compressing the extension

In order to load the extension on our automated browser we will first need to compress the Metamask extension to a .crx file, here are the steps:

  • Install Metamask on your regular chrome
  • Navigate to chrome://extensions/
  • Click 'Pack extension' and enter the local path to the Metamask extension This will generate a .crx file that you can use to load as an extension on Chromium. Save the name of the folder where the extension is installed, this will be the 'Extension ID' that we will use later.

Loading the extension

To load Chromium with Metamask installed run:

from selenium import webdriver

EXTENSION_PATH = 'ENTER THE PATH TO YOUR CRX FILE'
opt = webdriver.ChromeOptions()
opt.add_extension(EXTENSION_PATH)

driver = webdriver.Chrome(chrome_options=opt)
Enter fullscreen mode Exit fullscreen mode

Interacting with Metamask

In order to interact with the dapp and Metamask simultaneously, we will need to have multiple tabs in our Chromium, one for the target Dapp and another one for Metamask itself.

When Chromium starts it will have a welcome screen for the Metamask extension, which will prompt you to setup your wallet, this is an example code to import an existing wallet (You may need to update some steps depending on your Metamask version):

driver.find_element_by_xpath('//button[text()="Get Started"]').click()
driver.find_element_by_xpath('//button[text()="Import wallet"]').click()
driver.find_element_by_xpath('//button[text()="No Thanks"]').click()

# After this you will need to enter you wallet details

inputs = driver.find_elements_by_xpath('//input')
inputs[0].send_keys(SECRET_RECOVERY_PHRASE)
inputs[1].send_keys(NEW_PASSWORD)
inputs[2].send_keys(NEW_PASSWORD)
driver.find_element_by_css_selector('.first-time-flow__terms').click()
driver.find_element_by_xpath('//button[text()="Import"]').click()
driver.find_element_by_xpath('//button[text()="All Done"]').click()
Enter fullscreen mode Exit fullscreen mode

After this Metamask will be successfully set up in Chromium, ready to connect to a Dapp.
When you need to interact with Metamask again you will need to use it in a different tab, like so:

EXTENSION_ID = 'ENTER HERE THE EXTENSION ID THAT YOU SAVED EARLIER'

driver.execute_script("window.open('');")
driver.switch_to.window(driver.window_handles[1])
driver.get('chrome-extension://{}/popup.html'.format(EXTENSION_ID))
Enter fullscreen mode Exit fullscreen mode

With this Metamask will be opened in this new tab, ready to be interacted with.


That's it, now you're ready to start automating interactions with Dapps.
Follow me for more articles like this and if you have any questions feel free to leave them in the comments below.

Discussion (12)

Collapse
fedeblengio profile image
Federico Blengio

Thanks bro, it has helped me a lot, but I have a problem for days I am trying to solve it but I can not.

any ideas?

Collapse
ltmenezes profile image
Leonardo Teixeira Menezes Author

Hey Federico, good to know that this article is helping you, thanks for the feedback!
This kinds of issues might happen because the behaviour of the extension might differ depending on the metamask extension version.
In these cases the best approach is to use some python debugger (such as IPDB) to stop the code for you to find exactly what is different in your version, most times you only need to do some minor changes.
Just followed you, feel free to drop a DM if you still have the issue!

Collapse
fedeblengio profile image
Federico Blengio

here is my problem
prnt.sc/1oab8yn

Collapse
baotran810 profile image
baotran810 • Edited

Hi Federico,
I also got the issue "NoSuchElementException" because it couldn't locate the "Get started" button. While debugging, I found that it focused on the first page when I ran the script, so I switched to the first tab to close then it only focused on the MetaMask page then the "Get started" button was able to locate. Here is the script I used, I hope it would help you.

await driver.getAllWindowHandles().then(function (handles) {
driver.switchTo().window(handles[1]);
driver.close()
driver.switchTo().window(handles[0]);
});

Thread Thread
fedeblengio profile image
Federico Blengio

Thank you

Thread Thread
fedeblengio profile image
Federico Blengio

I have a question to ask about handles, you can send a dm?

Thread Thread
escaraptor09 profile image
Marc Ferri

How did you finally solved it?????

Collapse
escaraptor09 profile image
Marc Ferri

I got the same problem

Collapse
bacolen profile image
Raphael Bachofen • Edited

So helpful! Finally someone posted this...

Collapse
techiesworld01 profile image
TechiesWorld01

Hi Leonardo,
I am facing issue while installation of selenium through chrome driver.

ImportError: cannot import name 'webdriver' from partially initialized module 'selenium' (most likely due to a circular import) (C:\Users\Admin\Desktop\selenium.py)

Collapse
p12 profile image
P12522 Brasil

Awesome!!!

Collapse
agustinustheo profile image
Agustinus Theodorus

I needed this. Thank you very much!