Schleife für mehrere URLs

Question

Ich versuche, eine Web-Seite mit dem folgenden Code zu kratzen: -

import requests 
from bs4 import BeautifulSoup 

page = requests.get("http://www.realcommercial.com.au/sold/property-offices-retail-showrooms+bulky+goods-land+development-hotel+leisure-medical+consulting-other-in-vic/list-1?includePropertiesWithin=includesurrounding&activeSort=list-date&autoSuggest=true") 

soup = BeautifulSoup(page.content, 'html.parser') 
links = soup.find_all('a', attrs ={'class' :'details-panel'}) 
hrefs = [link['href'] for link in links] 

for urls in hrefs: 
    pages = requests.get(urls) 
    soup_2 =BeautifulSoup(pages.content, 'html.parser') 

    Date = soup_2.find_all('li', attrs ={'class' :'sold-date'}) 
    Sold_Date = [Sold_Date.text.strip() for Sold_Date in Date] 
    Address_1 = soup_2.find_all('p', attrs={'class' :'full-address'}) 
    Address = [Address.text.strip() for Address in Address_1] 

der obige Code wird in dem hrefs nur die Details von der ersten URL zurück.

['Mon 05-Jun-17'] ['261 Keilor Road, Essendon, Vic 3040'] 

muss ich die Schleife durch jede URL in hrefs laufen & von jeder URL in hrefs ähnliche Details zurück. Bitte schlagen Sie vor, was ich im obigen Code hinzufügen/bearbeiten sollte. Jede Hilfe würde sehr geschätzt werden.

Dank

Answer 1

Es verhält sich in korrekter Weise. Sie müssen die Informationen in einer externen Liste speichern und dann zurückgeben.

import requests 
from bs4 import BeautifulSoup 

page = requests.get("http://www.realcommercial.com.au/sold/property-offices-retail-showrooms+bulky+goods-land+development-hotel+leisure-medical+consulting-other-in-vic/list-1?includePropertiesWithin=includesurrounding&activeSort=list-date&autoSuggest=true") 

soup = BeautifulSoup(page.content, 'html.parser') 
links = soup.find_all('a', attrs ={'class' :'details-panel'}) 
hrefs = [link['href'] for link in links] 
Data = [] 
for urls in hrefs: 
    pages = requests.get(urls) 
    soup_2 =BeautifulSoup(pages.content, 'html.parser') 

    Date = soup_2.find_all('li', attrs ={'class' :'sold-date'}) 
    Sold_Date = [Sold_Date.text.strip() for Sold_Date in Date] 
    Address_1 = soup_2.find_all('p', attrs={'class' :'full-address'}) 
    Address = [Address.text.strip() for Address in Address_1] 
    Data.append(Sold_Date + Address) 
return Data 

Answer 2

Sie überschreiben Address und Sold_Date Objekte auf jeder Iteration:

# after assignment previous data will be lost 
Sold_Date = [Sold_Date.text.strip() for Sold_Date in Date] 
Address = [Address.text.strip() for Address in Address_1] 

Versuchen leer list s außerhalb der Schleife zu initialisieren und sie

import requests 
from bs4 import BeautifulSoup 

page = requests.get("http://www.realcommercial.com.au/sold/property-offices-retail-showrooms+bulky+goods-land+development-hotel+leisure-medical+consulting-other-in-vic/list-1?includePropertiesWithin=includesurrounding&activeSort=list-date&autoSuggest=true") 

soup = BeautifulSoup(page.content, 'html.parser') 
links = soup.find_all('a', attrs={'class': 'details-panel'}) 
hrefs = [link['href'] for link in links] 

addresses = [] 
sold_dates = [] 
for urls in hrefs: 
    pages = requests.get(urls) 
    soup_2 = BeautifulSoup(pages.content, 'html.parser') 

    dates_tags = soup_2.find_all('li', attrs={'class': 'sold-date'}) 
    sold_dates += [date_tag.text.strip() for date_tag in dates_tags] 
    addresses_tags = soup_2.find_all('p', attrs={'class': 'full-address'}) 
    addresses += [address_tag.text.strip() for address_tag in addresses_tags] 

gibt uns

verlängern

>>>sold_dates 
[u'Tue 06-Jun-17', 
u'Tue 06-Jun-17', 
u'Tue 06-Jun-17', 
u'Tue 06-Jun-17', 
u'Tue 06-Jun-17', 
u'Tue 06-Jun-17', 
u'Tue 06-Jun-17', 
u'Mon 05-Jun-17', 
u'Mon 05-Jun-17', 
u'Mon 05-Jun-17'] 
>>>addresses 
[u'141 Napier Street, Essendon, Vic 3040', 
u'5 Loupe Crescent, Leopold, Vic 3224', 
u'80 Ryrie Street, Geelong, Vic 3220', 
u'18 Boase Street, Brunswick, Vic 3056', 
u'130-186 Buckley Street, West Footscray, Vic 3012', 
u'223 Park Street, South Melbourne, Vic 3205', 
u'48-50 The Centreway, Lara, Vic 3212', 
u'14 Webster Street, Ballarat, Vic 3350', 
u'323 Nepean Highway, Frankston, Vic 3199', 
u'341 Buckley Street, Aberfeldie, Vic 3040']