Scrapy with selenium, webdriver failing to instantiate

By : pad
Source: Stackoverflow.com
Question!

I am trying to use selenium/phantomjs with scrapy and I'm riddled with errors. For example, take the following code snippet:

def parse(self, resposne):

    while True:
        try:
            driver = webdriver.PhantomJS()
            # do some stuff
            driver.quit()
            break
        except (WebDriverException, TimeoutException):
            try:
                driver.quit()
            except UnboundLocalError:
                print "Driver failed to instantiate"
            time.sleep(3)
            continue

A lot of the times the driver it seems it has failed to instantiate (so the driver is unbound, hence the exception), and I get the blurb (along with the print message I put in)

Exception AttributeError: "'Service' object has no attribute 'process'" in <bound method Service.__del__ of <selenium.webdriver.phantomjs.service.Service object at 0x7fbb28dc17d0>> ignored

Googling around, it seems everyone suggests updating phantomjs, which I have (1.9.8 built from source). Would anyone know what else could be causing this problem and a suitable diagnosis?

By : pad


Answers

Selenium version 2.44.0 on pypi needs the following patch in Service.__init__ of selenium.webdriver.common.phantomjs.service

self.process = None

I was thinking of submitting a patch but this already exists in the most recent version on google code.

By : pad


This video can help you solving your question :)
By: admin