An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does scale? | Download the latest release

More cool linkd - hey - why am I moderated?

rated by 0 users
Answered (Verified) This post has 1 verified answer | 1 Reply | 1 Follower

posted on Tue, Oct 27 2009 5:37 PM

Blogs aboutWeb Crawler

Featured Blog

Classification and clustering [part one]

Web content mining is an interesting and wide domain. Almost everyone can modify one or several modules from a simple web mining system (like the one we’re building) and create a hole different … more →

Teofil Achirei Ads by Google
Web Spider Software
Extract web content and metadata from websites into your database

Membuat Web Crawler Sederhana — 2 comments

Jimmi Kembaren wrote 1 week ago: Hi, kali ini saya akan mencoba menjelaskan bagaimana cara membuat web crawler sederhana. Oh ya, web … more →

Tags: crawlerCurlWeb

Cara cerdas membuat web crawler — 6 comments

Jimmi Kembaren wrote 3 weeks ago: Bayangkan indahnya bisa mengumpulkan semua informasi yang ada di dunia ini. Semua informasi yang ada … more →

Tags: crawlerhostingWeb

PHP Web crawler

ppshein wrote 2 months ago: We’re trying to create Search engine for Burmese websites. First of all, we need web crawler k … more →

Tags: INFORMATIONSphpBotsspider

Web Crawler knowledge hounds unite!

webmining wrote 2 months ago: Hello World! This site is for anyone that desires cutting edge knowledge on the latest trends, techn … more →

Tags: ANT'swormsBotscrawlerweb crawlersWeb spiderCrawler KnowledgespidersCrawler Trends

Search Engines: biases and problems — 5 comments

Benjamin Steele wrote 3 months ago: I had a recent post disappear from listing on Word Press and shortly after it disappeared almost ent … more →

Tags: BloggingSocioPoliticalBiasbiasesBlogGooglePostProblemsSearch Engine

Strong as Steel: Day and Night Home Fitness — 2 comments

schrati wrote 4 months ago: Bäh, ich will nicht mehr. Ich blogge jetzt, weil ich das Schriftbild meiner Bacherlorarbeitsquellen … more →

Tags: Catholic Dungeon - kath. Gruselkabinettcand. B.Sc. oec. troph.Food Watch

Web Page Crawling

seoamit8 wrote 4 months ago: SNV Infotech A Web crawler is a computer program that browses the World Wide Web in a methodical, au … more →

Tags: SEOseo servicesSEO Services India

Classification and clustering [part one] — 2 comments

Teofil Achirei wrote 5 months ago: Web content mining is an interesting and wide domain. Almost everyone can modify one or several modu … more →

Tags: Programmingweb miningbest-firstclassificationCluster AnalysisClusteringfocused crawlerK-MeansK-nearest neighbor

The Thesaurus

Teofil Achirei wrote 5 months ago: It’s time to make our focused web crawler aware about it’s topic: the thesaurus. The sim … more →

Tags: Programmingweb miningcrawlerfocused crawlerThesaurusWeb MiningWeb spider

The Link

Teofil Achirei wrote 5 months ago: Let’s come back to our simple focused crawler. It’s time to start filtering the links we … more →

Tags: Programmingweb miningcrawlerfocused crawlerWeb MiningWeb spider

Wütende Kunst

schrati wrote 5 months ago: Ich bin kürzlich durch eine H-Dozentin auf einen durchaus lesenswerten Blog gestoßen, dessen Stil mi … more →


schrati wrote 5 months ago: Ihr habt keine Zeit? Warum ersteigert ihr sich nicht bei Ebay? Jetzt Mitbieten und sofort Gottesdien … more →

Tags: Catholic Dungeon - kath. Gruselkabinettcuriós

Does your ISP block Web Crawling?

netequalizer wrote 6 months ago: By Art Reisman Editor’s note: Art Reisman is the CTO of APconnections. APconnections designs a … more →

Tags: bandwidth shaperblock web crawlingInternet RobotISP blocknetequalizerweb crawler exampleweb robot

Web Crawler Architectures — 1 comment

Teofil Achirei wrote 6 months ago: Let’s see some common web spider architectures: the big picture a basic web crawler and a larg … more →

Tags: Programmingweb miningArchitecturecrawcrawlercrawler architecturelarge scale web crawlerWeb

A Simple Serial Focused Web Crawler 10

Teofil Achirei wrote 7 months ago: Let’s see what we’ve covered so far: as long as there are addresses in URL Queue, repeat … more →

Tags: Programmingweb miningcrawlerfocused crawlerinformation retreivalWeb Mining

Errata - New URLs Queue

Teofil Achirei wrote 7 months ago: If you’ve heard about Agile Programming, Extreme Programming or you’ve been working on s … more →

Tags: Programmingweb miningWeb Mininginformation retreivalcrawlerWeb spiderfocused crawlerurl frontier

When I grow up, I want to be a search engine! — 2 comments

khromov wrote 7 months ago: To the outside world, the Majestic12 website doesn’t look like much – it hardly even res … more →

Tags: LifeComputersTechnical SolutionsGamesdevelopmentweird thingsthe cloudWindowsamerica

Search Engine is the 21st century infrastructure!

Keren Dagan wrote 7 months ago: Search is infrastructure Should we start looking at investments in building and maintaining search e … more →

Tags: MethodmonitoringobservationsSoftwareGoogleIBMMicrosoftnatural monopolySearch

A Simple Serial Focused Web Crawler 4

Teofil Achirei wrote 7 months ago: How the crawler works This pseudo algorithm shows how the crawler will work: as long as there are ad … more →

Tags: Programmingweb miningWeb Mininginformation retreivalcrawlerWeb spiderfocused crawler

Ads by Google
Data Mining Stock
The Only Stock you Need to Profit from the New Technology Revolution. Put Data Mining to Work
Learn how to bring analytic insight to operational systems & processes Internet Marketing Cert.
Learn E Commerce, SEO, SEM, Media & More! Learn 100% Online--Free Info Web Spider Software
Extract web content and metadata from websites into your database Web Crawler
Free Download Harvest Data today Web Crawler

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by

You are moderated for overrunning a CommunityServer SPAM value threshold.  Try reducing the number of links in any given post.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (2 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, LLC