Welcome to WuJiGu Developer Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
354 views
in Technique[技术] by (71.8m points)

pyspider怎么维护一个全局变量

我需要采集一个列表,最后把所有的列表的结果生成一个json字段。

但是我发现,pyspider没办法创建一个全局变量。

每个列表的url,callback过去的是时候,每callback一次,就刷新了一次那个全局变量,最后只能留下一个列表的结果

就像下面的代码,大概写的,每一个list的callback都会让dbObj初始化,然后在更新,而不是在之前的值基础上更新

应该怎么维护一个全局变量啊在pyspider

class Handler(BaseHandler):
    
    
    dbObj = {
        
    }
    def list_page(self,response):
        self.crawl('list1',callback=self.detail_page)
        self.crawl('list2',callback=self.detail_page)
        self.crawl('list3',callback=self.detail_page)
        self.crawl('list4',callback=self.detail_page)
        self.crawl('list5',callback=self.detail_page)
        self.crawl('list6',callback=self.detail_page)
        self.crawl('list7',callback=self.detail_page)
        self.crawl('list8',callback=self.detail_page)
        
    def tail_page(self,response):
        object['list'+str(i)] = response.text

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

用save吧?
self.crawl('list1',callback=self.detail_page, save={key: value})
detail_page里就有save中存的值了


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to WuJiGu Developer Q&A Community for programmer and developer-Open, Learning and Share
...