I am trying to extract data from a search box, you can see a good example on wikipedia
This is my code:
driver = webdriver.Firefox() driver.get(response.url) city = driver.find_element_by_id('searchInput') city.click() city.clear() city.send_keys('a') time.sleep(1.5) #waiting for ajax to load selen_html = driver.page_source #print selen_html.encode('utf-8') hxs = HtmlXPathSelector(text=selen_html) ajaxWikiList = hxs.select('//div[@class="suggestions"]') items= for city in ajaxWikiList: item=TestItem() item['ajax'] = city.select('/div[@class="suggestions-results"]/a/@title').extract() items.append(item) print items
Xpath expression is ok, I checked on a static page. If I uncomment the line that prints out scrapped html code the code for the box shows at the end of the file. But for some reason I can't extract data from it with the above code? I must miss something since I tried 2 different sources, wikipedia page is just another source where I can't get these data extracted. Any advice here? Thanks!