Project Detail
Multithreaded Java crawler
Multithreaded Java crawler is project number 357774
posted at Freelancer.com. Click here to post your own project.
Status: Cancelled
Selected Providers: -
Budget: $250-750
Created: 12/15/2008 at 10:05 EST
Bid Count: 25
Average Bid:
$ 403
12/19/2008 at 10:05 EST
Project Creator:
profseo08
Employer Rating: (No Feedback Yet)
Description
I need a multithreaded Java program that can be controlled from the command line interface. I must be able to control things like number of threads, timeouts, and max number of simultaneous requests to the same IP. The program must be fast and well tested and able to resume processing from the point it got to before a crash. It can use a maximum of 128 MB of memory and should extract info on a huge list (millions) of URLs. For every URL the following needs to be stored in the output file in one line:URL
IP address
HTTP response code
Resolving URL if response is 301 or 302
Size of document in bytes
Content of title-tag
Content of meta-description
Content of meta-keywords
Content of all heading tag
Number of words in the body-section striped from html code
Number of links
(+ some other data)
Error message if any
Additional information submitted:
12/16/2008 at 2:43 EST:
All textual data should be decoded and 'clean' before written to the output file in UTF-8. I think apache.org has some pretty good decoding packages that help determine the encoding of the source code.
The program needs only to deal with http -not https
The program should run on JRE version 1.5.
It will most likely be tested on Mac OS X and deployed on a Linux box with 4 or 8 cores. I expect to process around 1000 to 1500 URLs every minute when deployed and tuned.
The amount of data in and out of the program will not fit in memory. The program must read data from file and write to the output file as it runs. The input file will contain from 10.000.000 to 20.000.000 URLs
12/16/2008 at 8:55 EST:
This is a sample input file with 196 URLs
Additional files submitted:
Sample-urls.txt
|
Job Type |
|
Provider Rating
WHAT IS FREELANCER.COM?
Freelancer.com is a global outsourcing solution and freelance jobs website. Here you can find freelance coders, writers, programmers, designers, marketers and more. You can also outsource projects and save a lot of money in the process. Getting the best web design, professional programming, custom writing or affordable marketing has never been easier!
We have thousands of satisfied clients from every corner of the world. Would you like to outsource your next project? Would you like to find freelance jobs and make money online? Just sign up to get started!
Outsourcing jobs is the process of hiring an outside company or individual to complete your projects. We have created a safe environment for both service buyers and service providers via our secure payment system. You outsource the project and don't release any money until you're satisfied! Try outsourcing for free today! Don't forget to bookmark our homepage for your next project or job.
- Plendo Sweden AB © 2004-2009
- PRIVACY POLICY
- TERMS
- AFFILIATES
- API
- QUOTES
- CONTACT
Freelancer™ is a trademark of Ignition Networks Pty Limited (ACN 134 845 748)
Newly Outsourced Projects fed by 
Follow our Freelance Jobs on Twitter 
Add to your
Google search results