Project Detail

Webcrawler to save link relations between websites  

Webcrawler to save link relations between websites is project number 435091
posted at Freelancer.com. Click here to post your own project.

 

| More
Free Trial For New Buyers
 

Status:

Selected Providers: rmvarasu

Budget: $250-750

Created: 05/14/2009 at 17:17 EDT

Bid Count: 18

Average Bid:
$ 438

06/13/2009 at 17:17 EDT

Project Creator: crilla
Employer Rating: 10/1010/1010/1010/1010/1010/1010/1010/1010/1010/10 (1 reviews)

Bid On This Project
 

Description

i need a coder for an webcrawler that saves the link relations between urls / domains

the crawler need the following features:
- performant for billion saved link relations!!!
- crawl filter: just crawl specific top level domains / websites within an specific country (based on ip country db)
- follow redirect links and saves them too
- respect robots.txt, meta robots and rel attributes
- multiple threads & crawlers (on different machines)
- one mysql database for all crawlers (OR: if you want to use an other database -> api to get the data)
- linux compatible

its ok for me if you use opensource libaries and codes as long as the crawler is fast enough and it doesnt generate too much cpu load

Messages Posted:0 View project clarification board Post message on project clarification board

Bid On This Project
 

If you are the project creator or one of the bidders Log In for more options

 

400

5 days

05-15-2009 00:34 EDT

We can help in your project, please check PMB to see our related experience.

help

 

400

10 days

05-15-2009 00:00 EDT

_____________________________________ SERIOUS bid here, let us discuss details given by you, and start ASAP. Please see my excellent 46 reviews given by my clients all over the world. I have 8+ years experience. Thanks and Regards : TUHIN _____________________________________

help

 

350

15 days

05-15-2009 02:13 EDT

Hey pls view pmb

help

 

400

15 days

05-14-2009 23:46 EDT

Can be done well for you, Thanks and regards!

help

 

400

10 days

05-16-2009 01:33 EDT

We do have experienced professionals for java and .net and hence we will deliver quality work. Please refer pmp for more details.

help

 

400

15 days

06-10-2009 10:33 EDT

Hello! We have done this kind of Web Crawling development before. Also we are very keen on customer satisfaction. So you are getting good experienced developers at a fantastic bargain !!! Please check PMB. Thank You!

help

 

450

12 days

05-14-2009 22:23 EDT

Please check PM. Thanks

help

 

600

15 days

05-30-2009 07:35 EDT

Hi,please check PM.

help

 

450

7 days

05-22-2009 06:00 EDT

Hi, Thanks for giving an opportunity to place a BID.Further kindly go through PM. Regards, Bhavik

help

 

280

20 days

05-18-2009 14:05 EDT

Hello, please allow me to do the job.

help

 

750

20 days

05-17-2009 10:25 EDT

can do this for you.thanks

help

 

500

20 days

05-14-2009 19:18 EDT

(No Feedback Yet)

I already have a framework that does what you need. But I have questions before accepting this task. -"link relations": what does this mean ? -just crawl specific top level domains / websites within an specific country (based on ip country db): So you need the user to specify a country, and the crawler just scrap the websites hosted on servers located in this country ? I'm a full time java programmer so it's gonna take sometime, that's why I need 20 days.

help

 

400

12 days

05-14-2009 20:24 EDT

(No Feedback Yet)

Dear friend pls kindly view pmb for more details

help

 

250

7 days

05-14-2009 23:41 EDT

(No Feedback Yet)

Hi, I can create this project. Thanks

help

 

700

20 days

05-15-2009 14:48 EDT

(No Feedback Yet)

Please check your PMB! Thanks.

help

 

300

15 days

05-15-2009 20:45 EDT

(No Feedback Yet)

I am an experienced Java professional having a total of 6+ years of working experience with various projects and technologies. I hold a Bachelor’s degree in Electrical Engineering. I have worked in Product Development projects and also in Online based, Web based application projects with very strong hands on GUI with Swing and Search Engine developments. I am interseted in this project.

help

 

550

30 days

05-18-2009 13:39 EDT

(No Feedback Yet)

hi, i am interested, we can discuss further details.

help

 

300

5 days

05-28-2009 05:44 EDT

(No Feedback Yet)

I have used the open source library used developed buy CMU. It was slow but I tweaked it a bit before. I could continue that work again.

help


    Bid on this Project