arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

New to arachnode.net

rated by 0 users
Answered (Verified) This post has 1 verified answer | 3 Replies | 1 Follower

Top 100 Contributor
Male
4 Posts
w3hunter posted on Mon, Aug 31 2009 1:14 PM

Hi,

I am new to arachnode.net. I have a question before I go in deep with crawler.

Is it possible to integrate it with any other web application ?

Here is the scenario.

I have one web application developed using asp.net & sql server 2008. 

Now I want to crawl few websites through crawler and I want to store information like link, description, keywords etc in my application database.

I hope my question is understandable..

Please reply asap.

Thanks in advance.

Answered (Verified) Verified Answer

Top 10 Contributor
Male
101 Posts
Answered (Verified) Kevin replied on Mon, Aug 31 2009 2:00 PM
Verified by w3hunter

Welcome!  The quick answer is sure, you can integrate by having them use the same db.  Use AN for what it's good at - crawling and storing the particular type of data you need.  Then, review the db schema and pull out what you need for use in your app.

You can also go deeper.  Write custom plug-ins that do whatever work you need during the crawl and store that data in any custom db schema you might need.  Choose Lucene for your text indexing, or go with SQL Server's FTI.  And with the full source, take it to wherever you need it!

Hope that helps :)

 

All Replies

Top 10 Contributor
Male
101 Posts
Answered (Verified) Kevin replied on Mon, Aug 31 2009 2:00 PM
Verified by w3hunter

Welcome!  The quick answer is sure, you can integrate by having them use the same db.  Use AN for what it's good at - crawling and storing the particular type of data you need.  Then, review the db schema and pull out what you need for use in your app.

You can also go deeper.  Write custom plug-ins that do whatever work you need during the crawl and store that data in any custom db schema you might need.  Choose Lucene for your text indexing, or go with SQL Server's FTI.  And with the full source, take it to wherever you need it!

Hope that helps :)

 

Top 100 Contributor
Male
4 Posts

Hi,

Thanks for your reply.

I will dig into it and will be back with list of questions.

Please be ready to answer them.... :)

Thanks.

Top 10 Contributor
Male
101 Posts
Kevin replied on Wed, Sep 2 2009 2:54 PM

No problem hope things go well for you.  AN is a great product with lots of potential to take it wherever you need to.

Page 1 of 1 (4 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC