Feature Requests

Re: Handling content type application/download

arachnode.net — Sat, 16 Apr 2011 19:51:33 GMT

Yes, it would be nice if there were a 1 to 1 mapping of content-types to extensions, but as you know, this isn't the case. Oh, well - the internet works pretty well without that.

Re: Handling content type application/download

ucg — Fri, 15 Apr 2011 20:50:56 GMT

Thanks, that got it going. I had to add "application/download" to the content types table first.

One issue that I had to work around is that the Arachnode file is output with the extension set in the AllowedDataTypes column "FullTextIndexTypes" regardless of the type of data being downloaded. For my application I set it to ".dat". To get real output names I already have a plugin that generates a table that maps the Arachnode "hash" name to the Discovery URI (or "crawlRequest.WebClient.HttpWebResponse.ResponseUri" which more acurately reflects the returned file) for prost-crawl moving and renaming of data, it is a post request plugin that builds entires during the crawl. That had to be altered to record the file name given in the "Content-disposition" part of the header so the original data type would not be lost.

Like this...

WebHeaderCollection whc = (crawlRequest.WebClient.HttpWebResponse != null)? crawlRequest.WebClient.HttpWebResponse.Headers : null;
string localcontenttype = (whc != null)? whc.Get("Content-Type").ToLower() : null;
string localfilename = ((localcontenttype != null) && localcontenttype.Equals("application/download")) ? whc.Get("Content-Disposition") : null;
if(localfilename != null) localfilename = Regex.Replace(localfilename,".*?filename=[\\\"]?([^\\\"]*).*","$1");

Re: Handling content type application/download

arachnode.net — Fri, 08 Apr 2011 17:34:00 GMT

You need to add the appropriate row to the cfg.AllowedDataTypes table.

Handling content type application/download

ucg — Fri, 08 Apr 2011 16:10:36 GMT

In attempting to gather data from a number of sites our spider has encountered a site that returns pages with the content type "application/download". The normal browser response is to prompt the user to view or save the file. Currently it appears that Arachnode ignores this content type entirely. Are there any plans to address this? Any advice?