Scrapy:如何手动从spider_idle事件回调插入请求?

Scrapy:如何手动从spider_idle事件回调插入请求?,第1张

Scrapy:如何手动从spider_idle事件回调插入请求
class FooSpider(baseSpider):    def __init__(self, *args, **kwargs):        super(FooSpider, self).__init__(*args, **kwargs)        dispatcher.connect(self.dont_close_me, signals.spider_idle)    def dont_close_me(self, spider):        if spider != self: return        self.crawler.engine.crawl(self.create_request(), spider)        raise DontCloseSpider("..I prefer live spiders.")

2016年更新:

class FooSpider(baseSpider):    yet = False    @classmethod    def from_crawler(cls, crawler, *args, **kwargs):        from_crawler = super(FooSpider, cls).from_crawler        spider = from_crawler(crawler, *args, **kwargs)        crawler.signals.connect(spider.idle, signal=scrapy.signals.spider_idle)        return spider    def idle(self):        if not self.yet: self.crawler.engine.crawl(self.create_request(), self) self.yet = True


欢迎分享,转载请注明来源:内存溢出

原文地址:https://54852.com/zaji/5630664.html

(0)
打赏 微信扫一扫微信扫一扫 支付宝扫一扫支付宝扫一扫
上一篇 2022-12-15
下一篇2022-12-16

发表评论

登录后才能评论

评论列表(0条)

    保存