Scrapy – Call function when spider closed

Hello Wednesday,

Today at work I had the chance to play with Scrapy. It is quite fast and really easy to use.
I will write posts about this crawling framework later, but for the time being I will take a quick note about how to call a function when spider closed.

After you crawl all the items and wanna do something else. Simply use the closed function.


# -*- coding: utf-8 -*-
import scrapy

class TcvStockSpider(scrapy.Spider):
	def closed(self, reason):
           #do something

In their document, there is another way to achieve this but I myself think this is simple and easy to achieve way.

Git – Untrack pyc files from source control

Why do we need to do this? What is a pyc file?
Python automatically compiles script to compiled code before execute it. Doing this will help your script run more smooth. And because this is automatically generated files, there is no use to commit a pyc file to your project’s source control.

$ find . -name '*.pyc' | xargs -n 1 git rm --cached