WEB SCRAPING – how to access content rendered through javascript ng-binding?

All we need is an easy explanation of the problem, so here it is.

I have tried scraping all of the player stats from

https://www.easports.com/madden-nfl/player-ratings?i=1&s=ovr_rating:DESC&v=true&=undefined

by using Beautiful soup:
blogscraper.py

import requests
from bs4 import BeautifulSoup
from csv import writer

response = requests.get('https://www.easports.com/madden-nfl/player-ratings/?i=1&s=ovr_rating:DESC&v=true&=undefined')
soup = BeautifulSoup(response.text, 'html.parser')

posts = soup.find_all(class_='player_rating-value')
print(posts)

But it gives me an empty list, or if I tried using

posts = soup.find_all(class_='ratings-hub_database')
print(posts)

It only gives me the stats up until strength

I have read other solutions that says I need to obtain the XHR data from the “Network” tab but I don’t know how

How to solve :

I know you bored from this bug, So we are here to help you! Take a deep breath and look at the explanation of your problem. We have many solutions to this problem, But we recommend you to use the first method because it is tested & true method that will 100% work for you.

Method 1

Open the network tab and press F5 to refresh page. Then Ctrl + F to open search box and enter a player name and press Enter. Look at results returned and you will find the following which returns the stats as json

from bs4 import BeautifulSoup as bs
import requests

data = requests.get('https://www.easports.com/madden-nfl/ratings/service/data?entityType=madden19_player&filter=iteration:1&sort=ovr_rating:DESC,%20lastName:ASC&limit=25').json()
print(data)

You could write to csv (note that you need to add your own header ordering)

from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

data = requests.get('https://www.easports.com/madden-nfl/ratings/service/data?entityType=madden19_player&filter=iteration:1&sort=ovr_rating:DESC,%20lastName:ASC&limit=25').json()
df = pd.DataFrame(data['docs'])
df.to_csv(r'C:\Users\User\Desktop\Info.csv', sep=',', encoding='utf-8-sig',index = False )

Using csv

from bs4 import BeautifulSoup as bs
import requests, csv

data = requests.get('https://www.easports.com/madden-nfl/ratings/service/data?entityType=madden19_player&filter=iteration:1&sort=ovr_rating:DESC,%20lastName:ASC&limit=25').json()
with open("data.csv", "w", encoding="utf-8-sig", newline='') as csv_file:
    w = csv.writer(csv_file, delimiter = ",", quoting=csv.QUOTE_MINIMAL)
    w.writerow(list(data['docs'][0].keys()))
    for row in data['docs']:
        w.writerow(list(row.values()))

Note: Use and implement method 1 because this method fully tested our system.
Thank you 🙂

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply