BeautifulSoup html csv

Question

Guten Abend, ich habe BeautifulSoup verwendet, um einige Daten von einer Website zu extrahieren, wie folgt:

from BeautifulSoup import BeautifulSoup 
from urllib2 import urlopen 

soup = BeautifulSoup(urlopen('http://www.fsa.gov.uk/about/media/facts/fines/2002')) 

table = soup.findAll('table', attrs={ "class" : "table-horizontal-line"}) 

print table 

Das folgende Ausgabe gibt:

[<table width="70%" class="table-horizontal-line"> 
<tr> 
<th>Amount</th> 
<th>Company or person fined</th> 
<th>Date</th> 
<th>What was the fine for?</th> 
<th>Compensation</th> 
</tr> 
<tr> 
<td><a name="_Hlk74714257" id="_Hlk74714257">&#160;</a>£4,000,000</td> 
<td><a href="/pages/library/communication/pr/2002/124.shtml">Credit Suisse First Boston International </a></td> 
<td>19/12/02</td> 
<td>Attempting to mislead the Japanese regulatory and tax authorities</td> 
<td>&#160;</td> 
</tr> 
<tr> 
<td>£750,000</td> 
<td><a href="/pages/library/communication/pr/2002/123.shtml">Royal Bank of Scotland plc</a></td> 
<td>17/12/02</td> 
<td>Breaches of money laundering rules</td> 
<td>&#160;</td> 
</tr> 
<tr> 
<td>£1,000,000</td> 
<td><a href="/pages/library/communication/pr/2002/118.shtml">Abbey Life Assurance Company ltd</a></td> 
<td>04/12/02</td> 
<td>Mortgage endowment mis-selling and other failings</td> 
<td>Compensation estimated to be between £120 and £160 million</td> 
</tr> 
<tr> 
<td>£1,350,000</td> 
<td><a href="/pages/library/communication/pr/2002/087.shtml">Royal &#38; Sun Alliance Group</a></td> 
<td>27/08/02</td> 
<td>Pension review failings</td> 
<td>Redress exceeding £32 million</td> 
</tr> 
<tr> 
<td>£4,000</td> 
<td><a href="/pubs/final/ft-inv-ins_7aug02.pdf" target="_blank">F T Investment &#38; Insurance Consultants</a></td> 
<td>07/08/02</td> 
<td>Pensions review failings</td> 
<td>&#160;</td> 
</tr> 
<tr> 
<td>£75,000</td> 
<td><a href="/pubs/final/spe_18jun02.pdf" target="_blank">Seymour Pierce Ellis ltd</a></td> 
<td>18/06/02</td> 
<td>Breaches of FSA Principles ("skill, care and diligence" and "internal organization")</td> 
<td>&#160;</td> 
</tr> 
<tr> 
<td>£120,000</td> 
<td><a href="/pages/library/communication/pr/2002/051.shtml">Ward Consultancy plc</a></td> 
<td>14/05/02</td> 
<td>Pension review failings</td> 
<td>&#160;</td> 
</tr> 
<tr> 
<td>£140,000</td> 
<td><a href="/pages/library/communication/pr/2002/036.shtml">Shawlands Financial Services ltd</a> - formerly Frizzell Life &#38; Financial Planning ltd)</td> 
<td>11/04/02</td> 
<td>Record keeping and associated compliance breaches</td> 
<td>&#160;</td> 
</tr> 
<tr> 
<td>£5,000</td> 
<td><a href="/pubs/final/woodwards_4apr02.pdf" target="_blank">Woodward's Independent Financial Advisers</a></td> 
<td>04/04/02</td> 
<td>Pensions review failings</td> 
<td>&#160;</td> 
</tr> 
</table>] 

ich diese exportieren möchten in CSV unter Beibehaltung der Tabellenstruktur wie auf der Website angezeigt, ist das möglich und wenn ja wie?

Vielen Dank im Voraus für die Hilfe.

Answer 1

Hier ist eine grundlegende Sache, die Sie versuchen können. Dies macht die Annahme, dass die headers alle in den <th> Tags sind, und dass alle nachfolgenden Daten in den <td> Tags sind. Dies funktioniert in dem Fall, den Sie zur Verfügung stellen, aber ich bin mir sicher, dass Anpassungen notwendig sind, wenn andere Fälle :) Die allgemeine Idee ist, dass, sobald Sie Ihre table finden (hier find, um die erste zu ziehen), erhalten wir die headers durch Iteration durch alle th Elemente, speichern sie in einer Liste. Dann erstellen wir eine rows Liste, die Listen enthält, die den Inhalt jeder Zeile darstellen. Dies wird durch Auffinden aller td Elemente unter tr Tags und unter der text, Codierung in UTF-8 (von Unicode) bevölkert. Sie erhalten dann eine CSV öffnen, schreiben die headers zuerst und dann alle der rows, but using (Zeile für Zeile in den Zeilen, wenn Zeile) Schreiben `keine leeren Zeilen eliminieren):

In [117]: import csv 

In [118]: from bs4 import BeautifulSoup 

In [119]: from urllib2 import urlopen 

In [120]: soup = BeautifulSoup(urlopen('http://www.fsa.gov.uk/about/media/facts/fines/2002')) 

In [121]: table = soup.find('table', attrs={ "class" : "table-horizontal-line"}) 

In [122]: headers = [header.text for header in table.find_all('th')] 

In [123]: rows = [] 

In [124]: for row in table.find_all('tr'): 
    .....:  rows.append([val.text.encode('utf8') for val in row.find_all('td')]) 
    .....: 

In [125]: with open('output_file.csv', 'wb') as f: 
    .....:  writer = csv.writer(f) 
    .....:  writer.writerow(headers) 
    .....:  writer.writerows(row for row in rows if row) 
    .....: 

In [126]: cat output_file.csv 
Amount,Company or person fined,Date,What was the fine for?,Compensation 
" £4,000,000",Credit Suisse First Boston International ,19/12/02,Attempting to mislead the Japanese regulatory and tax authorities, 
"£750,000",Royal Bank of Scotland plc,17/12/02,Breaches of money laundering rules, 
"£1,000,000",Abbey Life Assurance Company ltd,04/12/02,Mortgage endowment mis-selling and other failings,Compensation estimated to be between £120 and £160 million 
"£1,350,000",Royal & Sun Alliance Group,27/08/02,Pension review failings,Redress exceeding £32 million 
"£4,000",F T Investment & Insurance Consultants,07/08/02,Pensions review failings, 
"£75,000",Seymour Pierce Ellis ltd,18/06/02,"Breaches of FSA Principles (""skill, care and diligence"" and ""internal organization"")", 
"£120,000",Ward Consultancy plc,14/05/02,Pension review failings, 
"£140,000",Shawlands Financial Services ltd - formerly Frizzell Life & Financial Planning ltd),11/04/02,Record keeping and associated compliance breaches, 
"£5,000",Woodward's Independent Financial Advisers,04/04/02,Pensions review failings,