using Scrapy for crawling ajax pages [closed]

By : hanu
Source: Stackoverflow.com
Question!

i am using Scrapy for a month. I am able to crawl and scrape several websites(infact i have crawled 900 websites) based on keywords given in pipelines.Now , the problem is when we come across javapages (ajax), scrapy doesnt scrape. I am trying to use following code selenium code for scraping ajax pages using scrapy and make suitable changes to scrape ajax pages

Do u guys have any good idea for scraping javapages using scrapy

second problem , i face is scrapy doesnt scrape login page

By : hanu


Answers

First monitor the Request that has been sent using ajax (use fiddler for this purpose or firefox firebug), then make similar Request in scrapy and it will bring the response.

and i cant understand what do u mean by "scrapy doesnt scrape login page": does it mean that u cant scrap pages that requires u to be logged in or the login page itself



This video can help you solving your question :)
By: admin