I am trying to scrape data off a website using Scrapy, a python framework. I can get the data from the website using the spiders but the problem occurs when I try to navigate through the website.
Also, as stated in the accepted answer, I cannot use mechanize or lxml. It suggests using a combination of
Selenium and Scrapy.
Function of the button:
I also was looking at
CasperJS and PhantomJS. Will they work?
I just need to automate the clicking of a button. How do I go about this?