arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

How to Convert the project to a WindowsApplication??

rated by 0 users
Answered (Verified) This post has 1 verified answer | 5 Replies | 2 Followers

Top 50 Contributor
7 Posts
wumengge posted on Tue, Apr 21 2009 2:27 AM

Until now  the arachnode.net  is a consoleApplication,and it is not easy to use for crawl the website,eg. we have to change the target URI by changing the souce code and compile the whole project once again.

So I decide to convert the project into a windowsApplication.Maybe it better to use.Is it easy to do that?

I just want to know which part of the code is necessary to rewrite,and which is the best way  to accomplish it.

Could I receive some suggestions to help me do it?

BTW: I have found a little bug of the consoleApplication,that is when the PC is off line,the consoleApplication will throw an exception and the process cannot be finished correctly.

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Sounds good.  Always glad to help.  And, please feel free to make suggestions.  We are open to what you think about arachnode.

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

All Replies

Top 10 Contributor
1,905 Posts

Try inserting rows into the CrawlRequests table in the database.  You won't need to recompile then.  :)   Tell me what you want to crawl and I will help you set the configuration to do so.

We are working on a web admin interface but don't have a timeframe for when it will be released.

If you are interested in creating a Windows app, start with wiring start, stop, pause and resume to the UI.  And, check out ArachnodeDAO.SubmitCrawlRequest.  Then, get back to me - maybe send me the code or post here and I'll take a look/give my input.  OK?

Great catch on the machine being offline.  I'll take a look and fix what I find...

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 50 Contributor
7 Posts

All right,I'll continue looking up the code,and send my problems to  you as  soon as  possible.

Thanks! : )

WuMengge

Top 10 Contributor
1,905 Posts
Verified by arachnode.net

Sounds good.  Always glad to help.  And, please feel free to make suggestions.  We are open to what you think about arachnode.

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 50 Contributor
7 Posts

I need your help now,I try to convert the IO of arachnode into a windowsForm.

I want to send my code to you,so could you give me your E-Mail address?

Top 10 Contributor
1,905 Posts

Wumengge - I send you a PM.  :)

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (6 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC