Supporting asyncio event loop

30 Jun 2018

This Blogpost deals with supporting asyncio in scrapy.

Supporting Asyncio event loop in scrapy

After the first evaluation, I did not have any talk with my mentor, so I went forward with my stipulated task of writing tasks for the code that I wrote. Ofcourse, as I stated earlier, I discussed with my mentor in writing the proposal,that I would learn writing tests, and would learn and adapt, while covering test suites for my program code. So I did go about my task. Halfway through, I fell sick, so it took 4–5 days to get ready, and covering up for the lost time. I had a discussion with my mentor, regarding the goals of proposal, asking opinions as to how I was going through. My mentor was pretty impressed with the work, so he asked me support other coroutines in scrapy. Till now, we were going well, so my mentor was intended in supporting other coroutines, which can be of good importance.

Preparing for the goal

While I had programmed using asyncio, I still needed some ideas, about going through the task, and to figure out as to what would be the plan, about charting out different libraries that I might need, and using asyncio,in scrapy. Asyncio is supported in Twisted, so we could use it in getting an overhead with the task, and designing the new code part that would be added in scrapy. As it is a new API, so it would be optional, enabling users to use as and when required.

Progress uptil now

While a lot of time has gone since the first evaluations, I would look forward in implementing the ideas, that I planned, and supporting other coroutines, is another stepping stone to completed my proposed goal, for gsoc. The upcomming week would be quite busy, in the sense that I need to cover up a lot, so would be looking forward to another period of intense coding.