Aktualisierungsliste, während der Iteration eingestellt - while-Schleife

Question

Ich habe ein scrapy Skript wie folgt

1) Sammelt navigation_path in eine Liste und ruft eine neue Parsing

g_next_page_list = [] 
g_next_page_set = set() 

def parse(self,response): 

    #code to extract nav_links 

    for nav_link in nav_links: 
     if nav_link not in g_next_page_set: 
      g_next_page_list.append(nav_link) 
      g_next_page_set.add(nav_link) 

    for next_page in g_next_page_list: 
     next_page = response.urljoin(next_page) 
     yield scrapy.Request(next_page, callback=self.parse_start_url, dont_filter=True,) 

und ich habe parse_start_url definiert als :

def parse_start_url(self,response): 

    #code to extract nav_links 

    for nav_link in nav_links: 
     if nav_link not in g_next_page_set: 
      g_next_page_list.append(nav_link) 
      g_next_page_set.add(nav_link) 

jedoch die globale Liste und an der Haupt Parse (g_next_page_set, g_next_page_list) nicht angehängt zu werden. Was mache ich falsch?

Vielen Dank im Voraus!

Answer 1

Sie verwenden keine globalen hier verwenden Sie self.variable_name

g_next_page_list = [] 
g_next_page_set = set() 

def parse(self,response): 

    #code to extract nav_links 

    for nav_link in nav_links: 
     if nav_link not in v_next_page_set: 
      self.g_next_page_list.append(nav_link) 
      self.g_next_page_set.add(nav_link) 

    for next_page in v_next_page_list: 
     next_page = response.urljoin(next_page) 
     yield scrapy.Request(next_page, callback=self.parse_start_url, dont_filter=True,) 


def parse_start_url(self,response): 

    #code to extract nav_links 

    for nav_link in nav_links: 
     if nav_link not in v_next_page_set: 
      self.g_next_page_list.append(nav_link) 
      self.g_next_page_set.add(nav_link) 

Dies sollte es