What is the proper way to control arachnode configuration?
We have different configuration tables in AN database. Still when I do crawler.Start() it tries to load disallowed domains, disallowed uris and some other config stuff from xml files on the disk.
I'm writing my console app for crawling, so I don't have any xmls with config. Could you give some advice, how to handle this properly?
Thank you :)
The .xml files should only be used if the DB cannot be found. Check your build directory.
For best service when you require assistance:
Figured out: the thing is to properly set up connection string in App.config file of a project that you run. Something like this:
connectionString="Data Source=LALALA-PC\SQLEXPRESS;Initial Catalog=arachnode.net;Integrated Security=True;Connection Timeout=3600; Max Pool Size=100000;"
Don't hardcode the connection string into your code.
The .xml configs are present if you're using one of the database overrides to simulate database presence.
Have you tried customizing Console\Program.cs - this file has everything you need?
The .xml files will be there if you use the DB, and then can easily be used for your non-DB applications.
Are you using the DEMO or the code from SVN?
Not sure, what overrides are you talking about. Is it overriding one of crawler.ApplicationSettings? Then which one is responsible for simulating db?
yea, actually I'm working on creating my own console crawler in my main project solution, taking AN Console app from SVN as a base. I'm connecting to the AN database, restored from .bak as it was written in installation guide. So I'm not sure, why is my console app looking for any .xml config files.
Sorry if any questions are really simple :)
Thanks a lot!
If you're using a non-database storage system, then, you'll need something to mimic the the configuration tables in SQL, thus, the .xml files.
hmm.. Could you maybe advice a place to debug it? because I have restored and reset the database, but still the app throws exception, if does not find xml files with configs
Screenshot(s), specific exception(s)?
This is what I get, when starting engine:
The database connection works well, because I get this exception written into the Exceptions table. Can't really figure out, where am I missing the point..
What is your ConnectionString, here?
Does it match here:
If dbo.DisallowedAbsoluteUris.xml isn't on disk, then it looks like you've never been able to connect to the DB to create the file.
Was this not clear in the instructions? (needs to be improved?)
If we speak about installation instructions, I think there is a space for improvement. What I faced, working in visual studio: when I add connection string to the project in 'properties' -> 'settings' tab, it silently adds kind of project's namespace to connection string name. So no way to set arachnode connection string this way. Only writing it yourself into App.config file.
The other thing to make clear is that you have to set connection string in App.config for the project that you currently build and run. Because ConfigurationManager, which is used in Configuration project will read this file for connection strings.
I had no notifications in visual studio about changing connection string, I also could not follow instruction for checking connection string for Functions project, because I see no 'database' tab in project's properties, I also could not find 'Functions.csproj.user' file. So it was all pretty much confusing for a not experienced .net user like me.
It is also good to mention that, if anyone uses arachnode dlls as a dependency for their own projects, they should also state connection string in App.config file of a project, that they build and run.
OK - I will take a look this weekend.
I am planning to update the DEMO (a good number of fixes and enhancements) to VS2015+.
Thank you for your feedback.
No problem, thank you too! Will you announce the update, when released?
Other things came up this weekend.
BTW, the DEMO update and instructions update won't affect you in any way... crawl away!