I need to code a bot that needs to do the following:
Go to a jsp page and search for something by:
1: writing something on a search box
2: clicking the search button(submit button)
3: clicking one of the the resulting buttons/links(same jsp page with different output)
4: get the entire html of the new page(same jsp page with different output)
The 4th one can be done with screen scraping and I do not think I need help with it. But I need some guidance to do the options from 1 to 3. Any links or just some keyword that will help me google to learn about it will be appreciated. I plan to do this with java.
java
screen-scraping
bots
Share
Improve this question
Follow
edited May 8 '12 at 10:06
alain.janinm
19k1010 gold badges5656 silver badges103103 bronze badges
asked Mar 16 '11 at 9:04
Tunca Ersoy
33911 gold badge44 silver badges1818 bronze badges
Add a comment
5 Answers
8
Maybe it's not what you want, but you can try selenium : http://seleniumhq.org/
It's a web application testing system.
Share
Improve this answer
Follow
answered Mar 16 '11 at 9:07
joel1di1
74155 silver badges1111 bronze badges
Add a comment
10
All you need is HTMLUnit
This is an extract from its description
HtmlUnit is a "GUI-Less browser for Java programs". It models HTML documents and provides an API that allows you to invoke pages, fill out forms, click links, etc... just like you do in your "normal" browser.
P.S.: Had used it to build a web scraping project ;)
Share
Improve this answer
Follow
answered Mar 16 '11 at 9:07
Clyde Lobo
8,74766 gold badges3434 silver badges5858 bronze badges
Oh yeah, HTMLUnit is the tool of choice in that use case, i also used it (through Groovy) for scraping the web. – Riduidel Mar 16 '11 at 9:12
and it is worth to mention that it is really annoying that it does not support css selectors but xPath.. -.- – tObi Jan 6 '15 at 4:22
Add a comment
0
You can use python-mechanize for this.
Share
Improve this answer
Follow
answered Aug 26 '13 at 18:19
RATHI
4,55177 gold badges3434 silver badges4444 bronze badges
Add a comment
0
Prerequistes:
Selenium API.
Mozilla Firefox (with firebug extension installed)
We can achieve launching of a browser,go to the particular web page ,search a keyword and analyse results by doing following
Launch web browser(driver.launch()(selenium)
Go to the particular webpage(driver.get("your web pager"))(selenium)
Identify the search box(get identifier by using fire bug(id,xml path.. etc)
Go to that box and write your search keyword (webelement.sendkeys("your keyword") and click on search button (webelement.click())(selenium)
Click on desired result by using identifiers and for next web page to load (selenium)
Share
Improve this answer
Follow
edited Apr 16 '16 at 6:37
Tasos K.
7,69977 gold badges3838 silver badges5757 bronze badges
answered Apr 16 '16 at 6:14
user3736869
111 bronze badge
the answer is not exhaustive. Please provide a sample code that has worked for you – bademba Aug 17 '16 at 14:09
Add a comment
0
I used selenium in chrome. If you want to use selenium you have to download from http://www.seleniumhq.org/download/ --- the latest version and implement in neatbeans or eclipse the jar files. (Selenium Client & WebDriver Language Bindings, Selenium Standalone Server) After this you have to download from google https://sites.google.com/a/chromium.org/chromedriver/ -- chrome driver also the latest version extract the file and save on your pc.
Share
Improve this answer
Follow
Comments
Post a Comment