Clicking, Typing, Hovering and Scrolling with Selenium Clicking Buttons Entering Text Moving the Mouse Scrolling Conclusion

So you’ve tried to scrape some data from the latest website, only to realize your current tool set of parsing HTML pages no longer suffices.

With the rise of AJAX, many of today’s websites (including the likes of Netflix and AirBnB) use React.js or similar frameworks to build interactive interfaces where the DOM itself is updated fluently based on user interactions. This contrasts with older methods of navigating to a new URL and making an additional HTTP request.

In these scenarios, older tools such as BeautifulSoup may not be enough.

Getting Started with Selenium

If you aren’t familiar with Selenium, it allows you to manipulate and control a web browser. 

Originally designed to help developers create and run tests of their websites, it is also incredibly useful for scraping those same sites.

To start, you’ll need to download the appropriate Selenium WebDriver for the web browser of your choice. Then after the standard `pip [or conda] install selenium`, you can turn to your favorite Python environment and start a web scraping session, being sure to specify the file path of the driver like this:

1
2
3

from selenium import webdriver
driver  = webdriver.Firefox(“path_to_driver”) #.Chrome for Chrome

After running this code, you’ll notice a new browser window opens up. Viola!

To request a web page, use:

1

driver.get('url')

By default, Selenium will wait for the page to finish loading before the `.get(‘url’)` method completes. Once loaded, you can make selections of relevant DOM elements using various driver methods including

`driver.find_element_by_tag()`,

`driver.find_element_by_class()`,

`driver.find_element_by_id()`,

driver.find_elements_by_tag()`,

`driver.find_elements_by_class()`, and

`driver.find_elements_by_id().

Methods with plural naming (elementS) will select all matching elements while the singular versions will select only the first matching element. In addition, you can continue making additional narrower selections of child elements of these selections.

For example, the following code snippet will select all paragraphs within the body of the page, and then the text from those paragraphs:

1
2
3

body = driver.find_element_by_tag_name('body')
paragraphs = body.find_elements_by_tag_name('p')
paragraphs_text = [p.text for p in paragraphs])

You can also extract other relevant information such as the urls from hyperlinks:

1
2

links = driver.find_elements_by_tag_name('a')
link_urls = [l.get_property('href') for l in links]

Clicking Buttons

Moving things forward, you’ll probably find the need to click some buttons on the page, whether it is to login, follow a link or submit a search field. To do this, simply make an appropriate selection of the element you wish to click like above and run `element.click()`. If you wished to click on the first link from the selection above, you could run:

1
2
3

first_link = links_in_paragraphs[0]
first_link.click()

Entering Text

Aside from clicking elements, you’ll also want to be able to enter text, whether it is a login field, search bar, or form field. This can be done with the `element.send_keys()` method. If you’re trying to simulate more human-like typing, this can be further improved by delaying consecutive keystrokes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

import numpy as np
import time
text = “What should I search for?”
for c in text:
    element.send_keys()
    time.sleep(np.random.uniform(low=.2 high=.3)

Moving the Mouse

Sometimes, you need more than just clicking buttons and typing text. A common example is dropdown menus. In order for the subheading to become visible and clickable, you must first move the mouse over the dropdown menu. 

Here again, the first key is identifying the element of interest that you need to hover over. Afterwards, you can make the appropriate selection of the now visible element of interest, whether it’s a link to click or data from a hover tooltip.

1

webdriver.ActionChains(driver).move_to_element(gmailLink).perform()

For even more possibilities, see the Selenium Mouse actions in detail Documentation . 

Scrolling

Another common action worth adding to your tool set is scrolling. This can be especially useful for sites that render additional DOM elements and information when the user scrolls to the bottom of the screen.

In order to scroll, you can pass some javascript to the browser via the Selenium driver.

Here’s how to scroll 100 pixels:

1

driver.execute_script("window.scrollBy(0, 100)")

Similarly, if you already have selected the element of interest but need it to be visible in order for pertinent information to be rendered to the DOM, you can calculate the amount of scroll needed to render it within the current browser window.

To do this, you must note the window height of the browser, where the top of the DOM is currently situated, and the position of the element of interest. Putting this all together, here’s a function that can scroll to a given element:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

def scroll_to_element(driver, element):
"""Mimics human scrolling behavior and will put the element with 70 pixels of the center of the current viewbox."""
window_height = driver.execute_script("return window.innerHeight")
start_dom_top = driver.execute_script("return document.documentElement.scrollTop")
element_location = element.location['y']
desired_dom_top = element_location - window_height/2 #Center It!
to_go = desired_dom_top - start_dom_top
cur_dom_top = start_dom_top
while np.abs(cur_dom_top - desired_dom_top) > 70:
    scroll = np.random.uniform(2,69) * np.sign(to_go)
    driver.execute_script("window.scrollBy(0, {})".format(scroll))
    cur_dom_top = driver.execute_script("return document.documentElement.scrollTop")
    time.sleep(np.abs(np.random.normal(0.0472, 0.003)))

Conclusion

Web scraping is a valuable skillset for data acquisition, allowing you to generate new datasets from information across the web.

While many tools exists, Selenium is indispensable, allowing you to launch and control a web browser to mimic human interactions. Such interactions include moving the mouse (virtually), scrolling, clicking on elements, and typing text into fields. With this, you’ll be able to scrape more diverse and complex websites across the web.