Sipios devient Theodo FinTech, l’entité experte Services Financiers du groupe Theodo
.png)
.png)
Share this post
The article covers scraping methods, recommends official APIs, and introduces tools like Jsoup and Selenium WebDriver. Featuring a financial data scraping demo, it underscores strategy comprehension. Find the full code on GitHub.
Please note that this article will guide you through the process of extracting data from a website. Always make sure its usage respects the law and the Terms of Service of the targeted websites. Often, doing so requires written agreement from the party. If an official API meets your needs, make sure to always use it instead.
Definition of web scraping Web scraping : Extracting data from a website to use it in a different context.
Also called web harvesting or web data extracting.
The web is full of resources. But the data is not always available through an API. When a website owner is not able to develop APIs to fit your needs but is ok for you to fetch it in any way you see fit, one of the solutions to overcome this restriction is to perform web scraping to extract the data you need.
Web pages are written in 3 main languages: HTML (HyperText Markup Language), CSS (Cascading Style Sheet), and JS(JavaScript). When your web browser access a URL, it downloads the page behind the requested URL. Each language has its own responsibility, detailed as follows :
The two types of scraping don’t support the same languages.
Downloading the source code of a page and parsing it.
This is called passive scraping because you cannot interact with the DOM. The main advantage of this method is the speed. Scraping is a slow method to get data from the internet. By performing passive scraping you reduce the number of operations to the strict minimum.
But performing the smallest number of operation also reduces your ability to interact with the web page, which can be restricting.
Taking control of a real web browser and using it to find data on a web page
To display a page, your web browser will perform a set of operations. Sometimes the data you want is displayed after the JavaScript is executed, which means that you need to do every operation a web browser will normally do to be able to get the data you want. In this case, you need active scraping.
Tip: Some website will detect that you are harvesting their data by performing passive scraping on their website. To protect their data, they can redirect you to another page which won’t contain the data you want. When it happens, you will need to use active scraping to be like a normal user of their website. (For more information about preventing web scraping https://github.com/JonasCz/How-To-Prevent-Scraping) This is the most elaborated way of scraping. To the eyes of the web server, your computer is like any of the page’s users. But this ability also implies bad sides : active scraping is generally very slow due to all the operations the computer has to perform before being ready to parse the web page.
A Github repo is available for this example at https://github.com/alexadrien/scrapingarticle
As an example, we are going to scrape financial data from a page that looks like that:
This page contains the financial performance per sector. The data you can see on the right side of the page is the data we are seeking for.
If you inspect the page, you will see that the page performs an HTTP GET request to get the data behind that list. If we really wanted the data behind that page, we could get the data by performing the same request, but the data is not always available from an API. For this we will not use the API.
This page is interesting to parse because the website implemented a security check to avoid people to harvest their data at high frequency, if they detect that your IP address is trying to get the HTML too often, they will redirect you to another page. We are going to use both passive and active scraping for the sake of the example.
Let’s first try to understand which scraping method we should use. The first step of every scraping strategy is to identify whether the data is displayed by executing JavaScript or if it is embedded in the HTML request. To do that, get to the page and disable JavaScript for this page.
As you can see, when the page is refreshed after the JavaScript is disabled, the donut chart does not load. The data on the right is still there, and if you click on the sector names, you can see that the data still loads with the page. This means that passive scraping is a good candidate.
Now there is a second test to perform in order to choose the correct scraping strategy. It is about security. Most websites will not allow you to scrape their website at high frequency. Loading webpage multiple times per second for several seconds is very suspicious and if you do, the web server will redirect you to a page where you need to prove you are a human.
A complete decision tree would be :
Now let’s describe how we are going to scrape this page. First, let’s take a look at the structure of the page.
Each sector name is a link to a new page where the sector’s section is expanded and exposes new and more specific data. We will need to visit each page and scrape the specific data for each sector. For the sake of the example, we are going to scrape all the sector’s links with Jsoup (passive scraping method) and then use Selenium (active scraping method) for the rest of the data.
One of the most recurrent operations when doing scraping is the search for the right HTML element where you can efficiently scrape the data. Let’s take a look at the DOM (Document Object Model) of the page and try to find a CSS query.
The best way I found to find efficiently the elements in the DOM is by right-clicking on the elements in the Chrome Developer tools window (in the “elements” tab) and select “Copy selector” in the “Copy” section. If you target a sector name in the DOM and copy the CSS selector, you find :
#content > div > div > div.sector__data > div > div > div.sector-chart-and-data__data-table > div > div > div.sector-data-table__container > div:nth-child(8) > div > div.sector-data-table__sector-name > a
Notice that at the end of the CSS selector is
div.sector-data-table__sector-name > a
That CSS selector means that we are searching for link elements inside a div that has the class name indicated above. So we are searching for HTML elements like this :
<div class="sector-data-table__sector-name">
<a href="#">My link</a>
</div>
It corresponds to the link elements we want. Guidelines to find the perfect CSS selector :
Let’s define some Java classes to help us store the data in a comprehensive way.
class Performance {
private String name;
private String value;
public Performance(String name, String value) {
this.name = name;
this.value = value;
}
// getters and setters
// ...
}
class Sector {
private String name;
private List performances;
public Sector(String name, List performances) {
this.name = name;
this.performances = performances;
}
// getters and setters
// ...
}
class ScrapedData {
private List sectors = new ArrayList();
public ScrapedData() {
}
// getters and setters
// ...
}
The first step in the Jsoup scraping method is to get the page behind a URL.
Document document = Jsoup.connect(url).get();
This statement will perform a request to the URL specified, get the source code of the page, and then parse it. The Document object is a Jsoup-specific object. The next step is to get all the sector links:
Elements linksToSector = document.select("a.sector-data-table__sector-name-link");
The select method will search in the HTML DOM for an object that matches the CSS query specified. The next step is to adapt the data we found:
ArrayList<String> stringLinksToSectors = new ArrayList<String>();
for (int i = 1; i < linksToSector.size(); i++) {
stringLinksToSectors.add(BASE_URL + linksToSector.get(i).attr("href"));
}
The code block will iterate over the links we found and prepend the beginning of each URL (as those links are internal to the website, they do not have the domain name, but we need it).
<a href="markets/sectors/telecommunication-services">
Telecommunication Services
</a>
This link does not contain the root domain name. This is why we need to append it.
We want to get rid of the “All Sectors” link, thus the iteration begins at i=1.
At that point, we have a List object with all the sector’s links.
To access the sector-specific data, we need to visit and scrape every link we discovered in the previous steps. In order to avoid loading the pages too fast which seems wrong to the web server, we will use the Selenium way of scraping.
The first step is to initialize the ChromeDriver.
System.setProperty("webdriver.chrome.driver", "chromedriver");
ChromeDriver driver = new ChromeDriver();
The first line set the location property of the ChromeDriver, it will indicate the program how to find the chromedriver file. This is a file required by Selenium to emulate a Chrome browser. The second line instantiates a ChromeDriver Java object.
The next step is to iterate over the links we found in the previous steps and access the page behind the URL.
for (int i = 0; i < stringLinksToSectors.size(); i++) {
String linkToCurrentSector = stringLinksToSectors.get(i);
driver.get(linkToCurrentSector);
}
Once each page is loaded, we can find the elements we want
for (int i = 0; i < stringLinksToSectors.size(); i++) {
String linkToCurrentSector = stringLinksToSectors.get(i);
driver.get(linkToCurrentSector);
String masterGroupName = driver.findElementByCssSelector(".sector-data-table__sector-row--selected a").getText();
List<WebElement> groupNames = driver.findElementsByCssSelector(".sector-data-table__industry-group-name");
List<WebElement> groupValues = driver.findElementsByCssSelector(".sector-data-table__industry-group-return");
}
driver.close();
The lines 4, 5 and 6 get the name of the sector which page the driver is currently on, and then all the sub-sectors name and value.
The last line shuts down the Chrome instance.
Now let’s use our Java classes Performance, Sector, and ScrapedData to store all the data we scraped
ScrapedData resultData = new ScrapedData();
System.setProperty("webdriver.chrome.driver", "chromedriver");
ChromeDriver driver = new ChromeDriver();
for (int i = 0; i < stringLinksToSectors.size(); i++) {
String linkToCurrentSector = stringLinksToSectors.get(i); driver.get(linkToCurrentSector);
String sectorName = driver.findElementByCssSelector(".sector-data-table__sector-row--selected a").getText();
List<WebElement> subSectorName = driver.findElementsByCssSelector(".sector-data-table__industry-group-name");
List<WebElement> subSectorValue = driver.findElementsByCssSelector(".sector-data-table__industry-group-return");
ArrayList<Performance> values = new ArrayList<Performance>();
for (int j = 0; j < subSectorName.size(); j++) {
values.add(
new Performance(
subSectorName.get(j).getText(),
subSectorValue.get(j).getText()
)
);
}
Sector currentFinancialCategory = new Sector(sectorName, values);
resultData.getSectors().add(currentFinancialCategory);
}
Once that the scraping operation is done, we need to store data somewhere. The solution you choose for this step depends on your usage of the data. For this example, we will write the scraped data in a JSON file. We will use the Jackson Java library to convert the data to JSON format. At the end of our Java function, when the data is available, we can add :
ObjectMapper mapper = new ObjectMapper();
mapper.writeValue(new File("results.json"), resultData);
You can find the complete code and the guide to run the scraper at https://github.com/alexadrien/scrapingarticle.