Publisher review:Ruby web crawler reads in a list of links from links.dat, it then picks out the ones it can easily spider This Ruby script reads in a list of links from links.dat, it then picks out the ones it can easily spider and gets a list of URLs from each page listed in links.dat.
Every new URL it finds will be added to newlinks.dat for later spidering by another bot running along side this one.
Ruby web crawler is a Ruby script for Snippets scripts design by lordrich.
It runs on following operating system: Windows / Linux.
Ruby web crawler reads in a list of links from links.dat, it then picks out the ones it can easily spider
Operating system:Windows / Linux
Ruby web crawler script details 
