Basics of Selenium
Introduction
I had always heard of “bots” on the web. Everybody seemed to want them banned from their website. They always just seemed like trouble — a kind of trouble that I found awfully intriguing. At that point in the story, I was familiar up to the point of knowing that you had to code a bot. And, prior to starting my programming journey, I had the incorrect impression that the building of one of these “bots” would not be in my future. Little did I know that only a month in, with only a very small semblance of coding familiarity, I would already understand the basics of a bot’s inner workings.
The same package that allows for you to dabble in bot development gives you the power to do so much more. From automating ticket booking to automating your finances, it covers a broad range of opportunities and provides incredible potential.
Let me introduce you to Selenium. And if you haven’t yet stepped into its warm and welcoming arms, let my do my best to coerce you into its boundless embrace.
Here, we will be going over the very basics of the package. From here, I highly encourage you to continue your education on this amazing package!
Installing Selenium
This process is more involved than most installations. That being said, the objectives are simple. Here are the only two requirements:
Install Selenium in Terminal
Just run this command:
Install a Web Driver for Selenium
You can use any mainstream browser to tackle this part of installation. That said, many people with tutorials seemed to be recommending Chrome. And, as that is my default browser anyway, that is what we will be covering here.
1. Find out which version of chrome you are running, which can be easily found be going to Chrome → About Google Chrome
2. Download the corresponding google chrome chromedriver at the following link: https://sites.google.com/a/chromium.org/chromedriver/downloads
You should see a similar thing on your page:
3. Unzip the downloaded file
Setting Up Selenium in Jupyter Notebooks
After importing selenium and webdriver, you will need to specify a path to the chromedriver that you downloaded. Assuming that you kept the chromedriver in Downloads and are using a mac, enter the following to set your webdriver:
Make sure to replace the the ‘user-name’ with your own individual username.
Basics of Selenium
Here, we will be going over some of the most basics tasks that you can perform with selenium.
Accessing Web Page
Access any webpage by replacing ‘url’ with any website url of your choosing
Clicking
Click through to a different page on the website with the following steps. Replace link with any link title on the page. For example: if a website has a page titled “About”, then replace “Link” with “About.”
Setting variable ‘elem’ equal to that element on the page:
Use .click( ) attribute to click on that page element:
Search Something in the Search Bar
Right click on the page and use ‘inspect’ to find the id, name, or class or the search bar. Ideally, you want the id. But, if that is not available, you can use the name. At a last resort, you may have to use the class of the search bar to access it. Selenium will grab the first element that fits the parameters given, so be careful with class. The code below demonstrates a case in which the search bar has an id. In the case that it does not, you can replace the “id” in find_element_by_id with “name” or “class.”
In order to proceed, you will have to import one more thing from selenium:
To type something into the search bar, replace search with whatever you want. The code below will enter “search” into the search bar:
To hit the ‘enter’ button:
To Avoid Malfunction
There is a common problem that you may run into when you are working with selenium. Let’s say you want selenium to input something into the search bar. You are not taking into account the fact that it takes a second or two for the page to load in the first place. Selenium may encounter some errors when it searches for an id, name, or class that hasn’t yet been loaded onto the page. Thankfully, there is a nice and clean solution to this problem.
In order to solve this issue, you need to import a couple more things:
Use the following try and except statement to get around this issue:
Let’s break this down:
- driver is the webdriver defined above
- 10 means that selenium will wait 10 seconds before it executes the driver.quit( ), which will close out of the browser. **You can alter this to fit your needs.
- By.ID can be changed to By.Class, By.Name, etc.
- ‘element’ would need to represent the, in this case, particular id that you are searching for
Another option (a much simpler one at that) for avoiding this problem is to run the following code. This code will tell selenium to wait a specified number of seconds before it can continue:
You might want to use the try and except statement in more complex sequences of instructions. This idea segways nicely into our final topic below.
Action Chains
Selenium actions allows you to predefine some set of actions to perform in sequence — a que of desired actions. In order to do this, you must import another package from selenium:
To create an action chain, you must create an instance of it. In the code below, that instance is defined as actions:
You can set up these actions as follows (replacing element with the element you would like to click):
To execute the sequence:
In this example, selenium will click on the specified object. You can mess around with this and create any sequence you like.
Conclusion
While these are just the basics of selenium, my hope is that your mind went wild with possibility. There are an infinite number of projects to be done using selenium, and it is a very important tool to possess as any programmer, let alone a data scientist!
