Parsing PHP-Daten von HTML-Webseite mit Jsoup

Question

Ich bin mir nicht ganz sicher, wie man diese Frage oder Titel so formuliert es hier geht es. Ich verwende jsoup, um eine Webseite zu analysieren (http://champion.gg/statistics/) und ich versuche, die Statistiken aus ihrer Tabelle mit diesem Code zu greifen.

public void connect(String url) { 
    try { 
     Document doc = Jsoup.connect(url).userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36").get(); 
     System.out.println(doc.toString()); 
     Element table = doc.select("table[class=table table-striped]").first(); 
     Element tbody = table.select("tbody").first(); 
     Iterator<Element> rows = tbody.select("tr").iterator(); 
     rows.forEachRemaining(row -> { 
      System.out.println(row.toString()); 
     }); 
    } catch(IOException exception) { 
     if(Settings.DEBUG) { 
      Program.LOGGER.log(Level.SEVERE, "There was an error reading the document with the supplied URL!", exception); 
     } 
     Program.alert("Error loading webpage!"); 
    } 
} 

und es produziert dieses Ergebnis

<tr ng-repeat="champion in filteredChampions = (championData | startsWith:search.title | filter:roleSort | orderBy:[order+sortExpression.sortBy,order+sortExpression.lastSortBy])"> 
<td class="rank">{{indexNumber($index, filteredChampions.length)}}</td> 
<td ng-class="{'selected-column':determineSelected('title')}"> <a href="/champion/{{champion.key}}/{{champion.role}}"> 
    <div class="tsm-tooltip tsm-angular-champion-tt" data-type="champions" data-name="{{champion.key}}" data-id="{{matchupData}}"> 
    <div class="matchup-champion {{champion.key}}"></div> 
    <span class="stat-champ-title">{{champion.title}}</span> 
    </div> </a> </td> 
<td class="stats-role-title" ng-class="{'selected-column':determineSelected('role')}">{{champion.role}}</td> 
<td ng-class="{'selected-column':determineSelected('winPercent')}"> <span ng-class="{'top-half': (champion.general.winPercent >= 50), 'bottom-half': (champion.general.winPercent < 50)}">{{champion.general.winPercent}}%</span> </td> 
<td ng-class="{'selected-column':determineSelected('playPercent')}">{{champion.general.playPercent}}%</td> 
<td ng-class="{'selected-column':determineSelected('banRate')}">{{champion.general.banRate}}%</td> 
<td ng-class="{'selected-column':determineSelected('experience')}">{{champion.general.experience}}</td> 
<td ng-class="{'selected-column':determineSelected('kills')}">{{champion.general.kills}}</td> 
<td ng-class="{'selected-column':determineSelected('deaths')}">{{champion.general.deaths}}</td> 
<td ng-class="{'selected-column':determineSelected('assists')}">{{champion.general.assists}}</td> 
<td ng-class="{'selected-column':determineSelected('largestKillingSpree')}">{{champion.general.largestKillingSpree}}</td> 
<td ng-class="{'selected-column':determineSelected('totalDamageDealtToChampions')}">{{champion.general.totalDamageDealtToChampions}}</td> 
<td ng-class="{'selected-column':determineSelected('totalDamageTaken')}">{{champion.general.totalDamageTaken}}</td> 
<td ng-class="{'selected-column':determineSelected('totalHeal')}">{{champion.general.totalHeal}}</td> 
<td ng-class="{'selected-column':determineSelected('minionsKilled')}">{{champion.general.minionsKilled}}</td> 
<td ng-class="{'selected-column':determineSelected('neutralMinionsKilledEnemyJungle')}">{{champion.general.neutralMinionsKilledEnemyJungle}}</td> 
<td ng-class="{'selected-column':determineSelected('neutralMinionsKilledTeamJungle')}">{{champion.general.neutralMinionsKilledTeamJungle}}</td> 
<td ng-class="{'selected-column':determineSelected('goldEarned')}">{{champion.general.goldEarned}}</td> 
<td ng-class="{'selected-column':determineSelected('overallPosition')}">{{champion.general.overallPosition}}</td> 
<td ng-class="{'selected-column':determineSelected('overallPositionChange')}"><span class="glyphicon" ng-class="{'glyphicon-arrow-up': (champion.general.overallPositionChange > 0), 'glyphicon-arrow-down': (champion.general.overallPositionChange < 0), 'same-position': (champion.general.overallPositionChange === 0)}">{{Math.abs(champion.general.overallPositionChange)}}</span></td> 
</tr> 

Anstatt nun das Ergebnis für die durchschnittliche Menge der Herstellung tötet eine bestimmte Champion hat es champion.general.kills im Ergebnis sagen, werde ich bekommen. Wie analysiere ich die Seite, so dass anstelle von champion.general.kills wird es ein tatsächliches Ergebnis wie 8 geben?

Answer 1

Wenn es darum geht, Daten aus einer Webseite zu extrahieren, müssen Sie dorthin gehen, wo sich die Daten befinden. In diesem Fall befinden sich die Daten immer noch auf der Webseite, was gut ist. Sie müssen das Skript-Tag mit den Daten abrufen und analysieren. Vorerst nimmt dieser Beispielcode es den Script-Tag am Index ist 11.

public static void main(String[] args) 
{ 
    try 
    { 
     Document doc = Jsoup 
       .connect("http://champion.gg/statistics/") 
       .userAgent(
         "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36") 
       .get(); 
     System.out.println(doc.toString()); 
     Elements table = doc.select("script"); 
     Element script = table.get(11); 
     parseText(script); 
    } 
    catch (IOException exception) 
    { 

    } 
} 

public static void parseText(Element script) 
{ 
    String text = ((DataNode) script.childNode(0)).toString().trim(); 
    int index = text.indexOf("_id"); 
    while (index > 0) 
    { 
     index += 6;// Beginning of value 
     int endQuote = text.indexOf("\"", index); 
     String id = text.substring(index, endQuote); 
     index = text.indexOf("\"key\":\"", endQuote); 
     endQuote = text.indexOf("\"", index + 8); 
     String key = text.substring(index, endQuote); 
     index = text.indexOf("\"kills\":", endQuote); 
     endQuote = text.indexOf(",", index); 
     String kills = text.substring(index, endQuote); 
     text = text.substring(endQuote); 
     index = text.indexOf("_id", index); 
     System.out.println(id + key + kills); 
    } 
} 

Ausgang:

5812965753fa9743395ee93a "Schlüssel": "Urgot" kills ": 6,47

5812965753fa9743395ee93b" Schlüssel ": "Aatrox" kills ": 5,8

5812965753fa9743395ee93d" Schlüssel ":" Galio "tötet": 4.58

5812965753fa9743395ee940 "Schlüssel": "Kled" tötet ": 7.3 ...

Answer 2

Ich fand die Antwort mit der Hilfe von ProgrammersBlock. Indem ich die Skriptdaten retrahierte, verwandelte ich sie von JSON in ein vollständiges Java-Objekt!

package com.databot.web.parser; 

import java.io.IOException; 
import java.io.StringReader; 
import java.util.ArrayList; 
import java.util.List; 
import java.util.logging.Level; 

import org.jsoup.Jsoup; 
import org.jsoup.nodes.DataNode; 
import org.jsoup.nodes.Document; 
import org.jsoup.nodes.Element; 
import org.jsoup.select.Elements; 

import com.databot.Program; 
import com.databot.Settings; 
import com.databot.champions.ChampionStats; 
import com.databot.champions.Champion; 
import com.google.gson.stream.JsonReader; 

public class WebParser { 

public void connect(String url) { 
    try { 
     Document doc = Jsoup.connect(url).userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36").get(); 
     Elements table = doc.select("script"); 
     Element script = table.get(11); 
     parseText(script); 
    } catch(IOException exception) { 
     if(Settings.DEBUG) { 
      Program.LOGGER.log(Level.SEVERE, "There was an error reading the document with the supplied URL!", exception); 
     } 
     Program.alert("Error loading webpage!"); 
    } 
} 

public void parseText(Element script) 
{ 
    String text = ((DataNode) script.childNode(0)).toString().substring(22).trim(); 
    System.out.println(text); 
    List<Champion> champions = new ArrayList<>(); 
    try { 
     JsonReader reader = new JsonReader(new StringReader(text)); 
     reader.setLenient(true); 
     reader.beginArray(); 
     while(reader.hasNext()) { 
      reader.beginObject(); 
       String id = "", key = "", role = "", title = ""; 
       ChampionStats stats = new ChampionStats(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0, 0); 
      while(reader.hasNext()) { 
       String name = reader.nextName(); 
       if(name.equalsIgnoreCase("_id")) { 
        id = reader.nextString(); 
       } else if(name.equalsIgnoreCase("key")) { 
        key = reader.nextString(); 
       } else if(name.equalsIgnoreCase("role")) { 
        role = reader.nextString(); 
       } else if(name.equalsIgnoreCase("title")) { 
        title = reader.nextString(); 
       } else if(name.equalsIgnoreCase("general")) { 
        double winPercent = 0, playPercent = 0, banRate = 0, experience = 0, kills = 0, deaths = 0, assists = 0, totalDamageDealtToChampions = 0, totalDamageTaken = 0, totalHeal = 0, largestKillingSpree = 0, minionsKilled = 0, neutralMinionsKilledTeamJungle = 0, neutralMinionsKilledEnemyJungle = 0, goldEarned = 0; 
        int overallPosition = 0, overallPositionChange = 0; 
         reader.beginObject(); 
         while(reader.hasNext()) { 
          String gName = reader.nextName(); 
          if(gName.equalsIgnoreCase("winPercent")) { 
           winPercent = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("playPercent")) { 
           playPercent = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("banRate")) { 
           banRate = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("experience")) { 
           experience = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("kills")) { 
           kills = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("deaths")) { 
           deaths = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("assists")) { 
           assists = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("totalDamageDealtToChampions")) { 
           totalDamageDealtToChampions = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("totalDamageTaken")) { 
           totalDamageTaken = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("totalHeal")) { 
           totalHeal = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("largestKillingSpree")) { 
           largestKillingSpree = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("minionsKilled")) { 
           minionsKilled = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("neutralMinionsKilledTeamJungle")) { 
           neutralMinionsKilledTeamJungle = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("neutralMinionsKilledEnemyJungle")) { 
           neutralMinionsKilledEnemyJungle = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("goldEarned")) { 
           goldEarned = reader.nextDouble(); 
          } else if(gName.equalsIgnoreCase("overallPosition")) { 
           overallPosition = reader.nextInt(); 
          } else if(gName.equalsIgnoreCase("overallPositionChange")) { 
           overallPositionChange = reader.nextInt(); 
          } else { 
           reader.skipValue(); 
          } 
         } 
         reader.endObject(); 
         stats = new ChampionStats(winPercent, playPercent, banRate, experience, kills, deaths, assists, totalDamageDealtToChampions, totalDamageTaken, totalHeal, largestKillingSpree, minionsKilled, neutralMinionsKilledTeamJungle, neutralMinionsKilledEnemyJungle, goldEarned, overallPosition, overallPositionChange); 
       } else { 
        reader.skipValue(); 
       } 
      } 
      reader.endObject(); 
      champions.add(new Champion(id, key, role, title, stats)); 
     } 
     reader.endArray(); 
     reader.close(); 
    } catch (Exception e) { 
     Program.alert("Error reading JSON data!"); 
     e.printStackTrace(); 
    } 
    champions.forEach(champion -> { 
     System.out.println(champion.toString()); 
    }); 
} 
} 

Das ist mein voller WebParser Klasse, wenn jemand interessiert ist, ich bin sicher, dass es einen besseren Weg, oder eine effizientere Art und Weise, dies zu schreiben, aber das ist, was ab jetzt für mich gearbeitet hat!