If you want to get the data used to plot the line chart on the below website, what would you do? You can click the button to download it. But, what if you don't want to do it manually? Can you use a script to scrape the data? Might not, as the detailed data is hidden. However, we can always try to find the hidden data source on such a website. I will explain how I found the hidden data source of the below line chart.
Inspect Website
Website inspection can find many secrets. To find the code related to the line chart, I first click the button Download. Then, I select (not click) the second option "CSV(data)". Afterward, I left clicked my mouse to show the "Inspect" option.
After clicking the "Inspect" option, I found the code related to the chart. The code is highlighted blue on the left. I clicked it and it started to download a file. At this point, I was sure that the link to download the line chart data is there.
Also, by moving my curse on the area, a web address showed up. I thought this is the address!
I coped the code to extract the web address. But, the copied content only includes the majority of the web address. I used the showing-up address as a reference to modify and below code, but it does not work.
By further inspecting the address, I found "amp;" is added behind each "&". I removed the "amp;". Then the web address starts to work.
There are many date parameters inside. Also, quite many parts can be removed and still give you a valid data downloading web address. A short version of the web address is, where dt1 and dt2 are the date parameters in the format of “yyyy-mm-dd”.
https://fred.stlouisfed.org/graph/fredgraph.csv?id=EFFR&cosd=dt2&coed=dt1
Comments