Today at work I had the chance to play with Scrapy. It is quite fast and really easy to use.
I will write posts about this crawling framework later, but for the time being I will take a quick note about how to call a function when spider closed.
After you crawl all the items and wanna do something else. Simply use the
# -*- coding: utf-8 -*- import scrapy class TcvStockSpider(scrapy.Spider): def closed(self, reason): #do something
In their document, there is another way to achieve this but I myself think this is simple and easy to achieve way.