TypeError: nicht unterstützte Operandentypen für +: 'NoneType' und 'str' in Scrapies dupefilter

Question

Ich benutze einen Scraper mit JOBDIR (cf. https://doc.scrapy.org/en/latest/topics/jobs.html), so dass Crawls pausiert und fortgesetzt werden können. Der Schaber hat sich erfolgreich für eine Weile laufen, aber jetzt, wenn ich die Spinne kriechen ich Protokolle, die mit dem folgenden Ende:

scraper_1 | 2017-06-21 14:53:10 [scrapy.middleware] INFO: Enabled item pipelines: 
scraper_1 | ['scrapy.pipelines.images.ImagesPipeline', 
scraper_1 | 'scrapy.pipelines.files.FilesPipeline'] 
scraper_1 | 2017-06-21 14:53:10 [scrapy.core.engine] INFO: Spider opened 
scraper_1 | 2017-06-21 14:53:10 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
scraper_1 | 2017-06-21 14:53:10 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 
scraper_1 | 2017-06-21 14:53:12 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.apkmirror.com/sitemap_index.xml> (referer: None) 
scraper_1 | 2017-06-21 14:53:13 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.apkmirror.com/sitemap_index.xml> (referer: None) 
scraper_1 | Traceback (most recent call last): 
scraper_1 | File "/usr/local/lib/python3.6/site-packages/scrapy/utils/defer.py", line 102, in iter_errback 
scraper_1 |  yield next(it) 
scraper_1 | GeneratorExit 
scraper_1 | 2017-06-21 14:53:13 [scrapy.core.engine] INFO: Closing spider (closespider_errorcount) 
scraper_1 | Exception ignored in: <generator object iter_errback at 0x7f4cc3a754c0> 
scraper_1 | RuntimeError: generator ignored GeneratorExit 
scraper_1 | 2017-06-21 14:53:13 [scrapy.statscollectors] INFO: Dumping Scrapy stats: 
scraper_1 | {'downloader/request_bytes': 306, 
scraper_1 | 'downloader/request_count': 1, 
scraper_1 | 'downloader/request_method_count/GET': 1, 
scraper_1 | 'downloader/response_bytes': 2498, 
scraper_1 | 'downloader/response_count': 1, 
scraper_1 | 'downloader/response_status_count/200': 1, 
scraper_1 | 'finish_reason': 'closespider_errorcount', 
scraper_1 | 'finish_time': datetime.datetime(2017, 6, 21, 14, 53, 13, 139012), 
scraper_1 | 'log_count/DEBUG': 26, 
scraper_1 | 'log_count/ERROR': 1, 
scraper_1 | 'log_count/INFO': 10, 
scraper_1 | 'memusage/max': 75530240, 
scraper_1 | 'memusage/startup': 75530240, 
scraper_1 | 'request_depth_max': 1, 
scraper_1 | 'response_received_count': 1, 
scraper_1 | 'scheduler/dequeued': 1, 
scraper_1 | 'scheduler/dequeued/disk': 1, 
scraper_1 | 'scheduler/enqueued': 1, 
scraper_1 | 'scheduler/enqueued/disk': 1, 
scraper_1 | 'spider_exceptions/GeneratorExit': 1, 
scraper_1 | 'start_time': datetime.datetime(2017, 6, 21, 14, 53, 10, 532154)} 
scraper_1 | 2017-06-21 14:53:13 [scrapy.core.engine] INFO: Spider closed (closespider_errorcount) 
scraper_1 | Unhandled error in Deferred: 
scraper_1 | 2017-06-21 14:53:13 [twisted] CRITICAL: Unhandled error in Deferred: 
scraper_1 | 
scraper_1 | 2017-06-21 14:53:13 [twisted] CRITICAL: 
scraper_1 | Traceback (most recent call last): 
scraper_1 | File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 517, in _oneWorkUnit 
scraper_1 |  result = next(self._iterator) 
scraper_1 | File "/usr/local/lib/python3.6/site-packages/scrapy/utils/defer.py", line 63, in <genexpr> 
scraper_1 |  work = (callable(elem, *args, **named) for elem in iterable) 
scraper_1 | File "/usr/local/lib/python3.6/site-packages/scrapy/core/scraper.py", line 183, in _process_spidermw_output 
scraper_1 |  self.crawler.engine.crawl(request=output, spider=spider) 
scraper_1 | File "/usr/local/lib/python3.6/site-packages/scrapy/core/engine.py", line 210, in crawl 
scraper_1 |  self.schedule(request, spider) 
scraper_1 | File "/usr/local/lib/python3.6/site-packages/scrapy/core/engine.py", line 216, in schedule 
scraper_1 |  if not self.slot.scheduler.enqueue_request(request): 
scraper_1 | File "/usr/local/lib/python3.6/site-packages/scrapy/core/scheduler.py", line 54, in enqueue_request 
scraper_1 |  if not request.dont_filter and self.df.request_seen(request): 
scraper_1 | File "/usr/local/lib/python3.6/site-packages/scrapy/dupefilters.py", line 53, in request_seen 
scraper_1 |  self.file.write(fp + os.linesep) 
scraper_1 | TypeError: unsupported operand type(s) for +: 'NoneType' and 'str' 
apkmirrorscrapercompose_scraper_1 exited with code 0 

Es scheint, wie der Fehler in dupefilters.py verursacht wird. Ich habe mir den Quellcode https://github.com/scrapy/scrapy/blob/master/scrapy/dupefilters.py angesehen, konnte aber bisher nicht herausfinden, was diesen Fehler verursacht hat. Irgendwelche Ideen?

aktualisieren

Hier sind einige weitere Details, wie die Spinne umgesetzt wird. Es ist ein SitemapSpider wie folgt:

import scrapy 
from scrapy.spiders import SitemapSpider 
from apkmirror_scraper.spiders.base_spider import BaseSpider 


class ApkmirrorSitemapSpider(SitemapSpider, BaseSpider): 
    name = 'apkmirror' 
    sitemap_urls = ['http://www.apkmirror.com/sitemap_index.xml'] 
    sitemap_rules = [(r'.*-android-apk-download/$', 'parse')] 

    custom_settings = { 
     'CLOSESPIDER_PAGECOUNT': 0, 
     'CLOSESPIDER_ERRORCOUNT': 1, 
     'CONCURRENT_REQUESTS': 32, 
     'CONCURRENT_REQUESTS_PER_DOMAIN': 16, 
     'TOR_RENEW_IDENTITY_ENABLED': True, 
     'TOR_ITEMS_TO_SCRAPE_PER_IDENTITY': 50, 
     'FEED_URI': '/scraper/apkmirror_scraper/data/apkmirror.json', 
     'FEED_FORMAT': 'json', 
     'DUPEFILTER_CLASS': 'apkmirror_scraper.dupefilters.URLDupefilter', 
    } 

    download_timeout = 60 * 15.0  # Allow 15 minutes for downloading APKs 

    def start_requests(self): 
     for url in self.sitemap_urls: 
      yield scrapy.Request(url, self._parse_sitemap, dont_filter=True) 

wo die parse Methode in der BaseSpider Klasse definiert ist. Ich habe eine benutzerdefinierte URLDupefilter wie folgt definiert:

from scrapy.dupefilters import RFPDupeFilter 


class URLDupefilter(RFPDupeFilter): 
    def request_fingerprint(self, request): 
     '''Simply use the URL as fingerprint. (Scrapy's default is a hash containing the request's canonicalized URL, method, body, and (optionally) headers). Omit sitemap pages, which end with ".xml".''' 
     if not request.url.endswith('.xml'): 
      return request.url 

    def request_seen(self, request): 
     '''Same as the RFPDupeFilter's request_seen method, except that a fingerprint of "None" is viewed as 'not seen' (cf. https://stackoverflow.com/questions/44370949/is-it-ok-for-scrapys-request-fingerprint-method-to-return-none).''' 
     fp = self.request_fingerprint(request) 
     if fp is None: 
      return False       # These two lines are added to the original 
     if fp in self.fingerprints: 
      return True 
     self.fingerprints.add(fp) 
     if self.file: 
      self.file.write(fp + os.linesep) 

jedoch der Fehler aus zu sein scheint Scrapy Kommen eingebaute in RFPDupeFilter Klasse. Ich verstehe nicht, warum das immer noch aktiviert ist, wenn ich DUPEFILTER_CLASS auf die benutzerdefinierte setzen?

Answer 1

class URLDupefilter(RFPDupeFilter): 
    def request_fingerprint(self, request): 
     '''Simply use the URL as fingerprint. (Scrapy's default is a hash containing the request's canonicalized URL, method, body, and (optionally) headers). Omit sitemap pages, which end with ".xml".''' 
     if not request.url.endswith('.xml'): 
      return request.url 

wird diese None zurück, wenn Anfrage-URL mit .xml endet. Dann versucht dupefilter, eine Zeichenfolge in eine Datei zu schreiben, indem er diese Zeichenfolge mit einem Zeilenumbruchzeichen kombiniert. Da es versucht, None und eine Zeichenfolge zu kombinieren, erhalten Sie TypeError.

Um dies zu beheben einfach eine Zeichenfolge für .xml Seiten zurück:

def request_fingerprint(self, request): 
     if not request.url.endswith('.xml'): 
      return request.url 
     return ''