Project Detail

Description

I am an academic researcher examining news for public companies. I have successfully created perl scripts (although quite crude and I am very new to this) to scrape data from websites before but this one is stumping me. I want to be able to take a list of search terms (a text file input with the stock exchange and ticker of each firm (e.g. NYSE WMT), select all news wires, news papers, and press releases (or each one of these at a time), and submit the search on www.highbeam.com/advancedSearch.aspx. This is the part I really need help with, but to finish the code I want to then take the search results and collect all of the dates of the articles (and headlines would be a bonus) and output the tickers, exchanges, headlines, and dates for all of the results. This is the free service part of highbeam.com, but if it works I am considering purchasing a subscription. So the input would be a list that would look something like:
NYSE WMT
NYSE IBM
etc.

and the output would look something like this:
Titles: EXCHG TICKER SOURCE DATE HDLN
DATA: NYSE WMT NEWSWR 01AUG2007 Bla bla bla
NYSE WMT NEWSWR 30AUG2006 Bla bla bla 2
NYSE WMT PRESSRL 01AUG2007 Bla bla bla 3
NYSE IBM NEWSWR 01AUG2007 Bla bla bla 4
etc.

Of course adding in anything to be courteous and not bog down their server would be nice too.

I have done similar things fairly easily with perl for sites that don't use post and javascript, but I can't quite figure this one out...

So complete, or even partial help would be very nice!