arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop

Completely Open Source @ GitHub

Does arachnode.net scale? | Download the latest release

Exception: The remote server returned an error: (406) Not Acceptable.

rated by 0 users
Not Answered This post has 0 verified answers | 5 Replies | 2 Followers

Top 10 Contributor
229 Posts
megetron posted on Mon, Aug 17 2009 1:25 PM

I am gettting 406 exception when try to get an image. why is that?

AbsoluteUri1
http://10net.co.il/108752/%D7%A6%D7%A4%D7%99%D7%99%D7%94-%D7%99%D7%A9%D7%99%D7%A8%D7%94-%D7%91%D7%A1%D7%A8%D7%98%D7%99-%D7%93%D7%A8%D7%9E%D7%94

AbsoluteUri2
http://10net.co.il/image/users/108752/ftp/my_files/%D7%A1%D7%A8%D7%98-%D7%93%D7%A8%D7%9E%D7%94.jpg

Message
The remote server returned an error: (406) Not Acceptable.

Source
System

StackTrace
   at System.Net.HttpWebRequest.GetResponse()     at System.Net.WebClient.GetWebResponse(WebRequest request)     at Arachnode.SiteCrawler.Components.WebClient.GetWebResponse(WebRequest request)

 

All Replies

Top 10 Contributor
1,905 Posts

This is due to certain WebServers not handling the HttpRequest header 'Accept-Types' properly.

You really are going to make me fix all of my little things, huh?  Stick out tongue

I'll take a look...

Just for a little background on why you may get this error.  Arachnode.net uses a very specific set of configurables to determine what to download, so you know exactly what what you are downloading.  A full set of ContentTypes is provided in cfg.ContentTypes.

In order to download a piece of content you will need to make an entry in cfg.AllowedDataTypes.  The most common internet DataTypes are already present in the default configuration.

There are some notable exceptions: Movies (huge), and JavaScripts (possibly malicious)..., and other scripts (also, possibly malicious)...

The FullTextIndexType column tells SQL and the iFilters (if installed) how to treat the source byte array when using SQL Full Text Indexing, if enabled.  Overrides allow the FullTextIndexType column to be more specific, should your iFilters require additional specificity.  (that was slightly redundant...!)

Here is where the 'Accept' HttpRequest header is assigned (WebClient.cs):

A list of all arachnode.net supplied Content-Types:

1 0 UNKNOWN
2 1 application/activemessage
3 1 application/andrew-inset
4 1 application/applefile
5 1 application/atom+xml
6 1 application/atomicmail
7 1 application/atomcat+xml
8 1 application/atomsvc+xml
9 1 application/auth-policy+xml
10 1 application/batch-SMTP
11 1 application/beep+xml
12 1 application/cals-1840
13 1 application/ccxml+xml
14 1 application/cea-2018+xml
15 1 application/cellml+xml
16 1 application/cnrp+xml
17 1 application/commonground
18 1 application/conference-info+xml
19 1 application/cpl+xml
20 1 application/csta+xml
21 1 application/CSTAdata+xml
22 1 application/cybercash
23 1 application/davmount+xml
24 1 application/dca-rft
25 1 application/dec-dx
26 1 application/dialog-info+xml
27 1 application/dicom
28 1 application/dns
29 1 application/dvcs
30 1 application/ecmascript
31 1 application/EDI-Consent
32 1 application/EDIFACT
33 1 application/EDI-X12
34 1 application/emma+xml
35 1 application/epp+xml
36 1 application/eshop
37 1 application/example
38 1 application/fastinfoset
39 1 application/fastsoap
40 1 application/fits
41 1 application/font-tdpfr
42 1 application/H224
43 1 application/http
44 1 application/hyperstudio
45 1 application/ibe-key-request+xml
46 1 application/ibe-pkg-reply+xml
47 1 application/ibe-pp-data
48 1 application/iges
49 1 application/im-iscomposing+xml
50 1 application/index
51 1 application/index.cmd
52 1 application/index.obj
53 1 application/index.response
54 1 application/index.vnd
55 1 application/iotp
56 1 application/ipp
57 1 application/isup
58 1 application/javascript
59 1 application/json
60 1 application/kpml-request+xml
61 1 application/kpml-response+xml
62 1 application/lost+xml
63 1 application/mac-binhex40
64 1 application/macwriteii
65 1 application/marc
66 1 application/mathematica
67 1 application/mbms-associated-procedure-description+xml
68 1 application/mbms-deregister+xml
69 1 application/mbms-envelope+xml
70 1 application/mbms-msk-response+xml
71 1 application/mbms-msk+xml
72 1 application/mbms-protection-description+xml
73 1 application/mbms-reception-report+xml
74 1 application/mbms-register-response+xml
75 1 application/mbms-register+xml
76 1 application/mbms-user-service-description+xml
77 1 application/mbox
78 1 application/media_control+xml
79 1 application/mediaservercontrol+xml
80 1 application/mikey
81 1 application/moss-keys
82 1 application/moss-signature
83 1 application/mosskey-data
84 1 application/mosskey-request
85 1 application/mpeg4-generic
86 1 application/mpeg4-iod
87 1 application/mpeg4-iod-xmt
88 1 application/mp4
89 1 application/msword
90 1 application/mxf
91 1 application/nasdata
92 1 application/news-transmission
93 1 application/nss
94 1 application/ocsp-request
95 1 application/ocsp-response
96 1 application/octet-stream
97 1 application/oda
98 1 application/oebps-package+xml
99 1 application/ogg
100 1 application/parityfec
101 1 application/patch-ops-error+xml
102 1 application/pdf
103 1 application/pgp-encrypted
104 1 application/pgp-keys
105 1 application/pgp-signature
106 1 application/pidf+xml
107 1 application/pidf-diff+xml
108 1 application/pkcs10
109 1 application/pkcs7-mime
110 1 application/pkcs7-signature
111 1 application/pkix-cert
112 1 application/pkixcmp
113 1 application/pkix-crl
114 1 application/pkix-pkipath
115 1 application/pls+xml
116 1 application/poc-settings+xml
117 1 application/postscript
118 1 application/prs.alvestrand.titrax-sheet
119 1 application/prs.cww
120 1 application/prs.nprend
121 1 application/prs.plucker
122 1 application/rdf+xml
123 1 application/qsig
124 1 application/reginfo+xml
125 1 application/relax-ng-compact-syntax
126 1 application/remote-printing
127 1 application/resource-lists-diff+xml
128 1 application/resource-lists+xml
129 1 application/riscos
130 1 application/rlmi+xml
131 1 application/rls-services+xml
132 1 application/rtf
133 1 application/rtx
134 1 application/samlassertion+xml
135 1 application/samlmetadata+xml
136 1 application/sbml+xml
137 1 application/scvp-cv-request
138 1 application/scvp-cv-response
139 1 application/scvp-vp-request
140 1 application/scvp-vp-response
141 1 application/sdp
142 1 application/set-payment
143 1 application/set-payment-initiation
144 1 application/set-registration
145 1 application/set-registration-initiation
146 1 application/sgml
147 1 application/sgml-open-catalog
148 1 application/shf+xml
149 1 application/sieve
150 1 application/simple-filter+xml
151 1 application/simple-message-summary
152 1 application/simpleSymbolContainer
153 1 application/slate
154 1 application/smil (OBSOLETE)
155 1 application/smil+xml
156 1 application/soap+fastinfoset
157 1 application/soap+xml
158 1 application/sparql-query
159 1 application/sparql-results+xml
160 1 application/spirits-event+xml
161 1 application/srgs
162 1 application/srgs+xml
163 1 application/ssml+xml
164 1 application/timestamp-query
165 1 application/timestamp-reply
166 1 application/tve-trigger
167 1 application/ulpfec
168 1 application/vemmi
169 1 application/vnd.3gpp.bsf+xml
170 1 application/vnd.3gpp.pic-bw-large
171 1 application/vnd.3gpp.pic-bw-small
172 1 application/vnd.3gpp.pic-bw-var
173 1 application/vnd.3gpp.sms
174 1 application/vnd.3gpp2.bcmcsinfo+xml
175 1 application/vnd.3gpp2.sms
176 1 application/vnd.3gpp2.tcap
177 1 application/vnd.3M.Post-it-Notes
178 1 application/vnd.accpac.simply.aso
179 1 application/vnd.accpac.simply.imp
180 1 application/vnd.acucobol
181 1 application/vnd.acucorp
182 1 application/vnd.adobe.xdp+xml
183 1 application/vnd.adobe.xfdf
184 1 application/vnd.aether.imp
185 1 application/vnd.airzip.filesecure.azf
186 1 application/vnd.airzip.filesecure.azs
187 1 application/vnd.americandynamics.acc
188 1 application/vnd.amiga.ami
189 1 application/vnd.anser-web-certificate-issue-initiation
190 1 application/vnd.antix.game-component
191 1 application/vnd.apple.installer+xml
192 1 application/vnd.arastra.swi
193 1 application/vnd.audiograph
194 1 application/vnd.autopackage
195 1 application/vnd.avistar+xml
196 1 application/vnd.blueice.multipass
197 1 application/vnd.bluetooth.ep.oob
198 1 application/vnd.bmi
199 1 application/vnd.businessobjects
200 1 application/vnd.cab-jscript
201 1 application/vnd.canon-cpdl
202 1 application/vnd.canon-lips
203 1 application/vnd.cendio.thinlinc.clientconf
204 1 application/vnd.chemdraw+xml
205 1 application/vnd.chipnuts.karaoke-mmd
206 1 application/vnd.cinderella
207 1 application/vnd.cirpack.isdn-ext
208 1 application/vnd.claymore
209 1 application/vnd.clonk.c4group
210 1 application/vnd.commerce-battelle
211 1 application/vnd.commonspace
212 1 application/vnd.cosmocaller
213 1 application/vnd.contact.cmsg
214 1 application/vnd.crick.clicker
215 1 application/vnd.crick.clicker.keyboard
216 1 application/vnd.crick.clicker.palette
217 1 application/vnd.crick.clicker.template
218 1 application/vnd.crick.clicker.wordbank
219 1 application/vnd.criticaltools.wbs+xml
220 1 application/vnd.ctc-posml
221 1 application/vnd.ctct.ws+xml
222 1 application/vnd.cups-pdf
223 1 application/vnd.cups-postscript
224 1 application/vnd.cups-ppd
225 1 application/vnd.cups-raster
226 1 application/vnd.cups-raw
227 1 application/vnd.curl
228 1 application/vnd.cybank
229 1 application/vnd.data-vision.rdz
230 1 application/vnd.denovo.fcselayout-link
231 1 application/vnd.dir-bi.plate-dl-nosuffix
232 1 application/vnd.dna
233 1 application/vnd.dpgraph
234 1 application/vnd.dreamfactory
235 1 application/vnd.dvb.esgcontainer
236 1 application/vnd.dvb.ipdcdftnotifaccess
237 1 application/vnd.dvb.ipdcesgaccess
238 1 application/vnd.dvb.iptv.alfec-base
240 1 application/vnd.dvb.iptv.alfec-enhancement
241 1 application/vnd.dvb.notif-aggregate-root+xml
242 1 application/vnd.dvb.notif-container+xml
243 1 application/vnd.dvb.notif-generic+xml
244 1 application/vnd.dvb.notif-ia-msglist+xml
245 1 application/vnd.dvb.notif-ia-registration-request+xml
246 1 application/vnd.dvb.notif-ia-registration-response+xml
247 1 application/vnd.dvb.notif-init+xml
248 1 application/vnd.dxr
249 1 application/vnd.dynageo
250 1 application/vnd.ecdis-update
251 1 application/vnd.ecowin.chart
252 1 application/vnd.ecowin.filerequest
253 1 application/vnd.ecowin.fileupdate
254 1 application/vnd.ecowin.series
255 1 application/vnd.ecowin.seriesrequest
256 1 application/vnd.ecowin.seriesupdate
257 1 application/vnd.emclient.accessrequest+xml
258 1 application/vnd.enliven
259 1 application/vnd.epson.esf
260 1 application/vnd.epson.msf
261 1 application/vnd.epson.quickanime
262 1 application/vnd.epson.salt
263 1 application/vnd.epson.ssf
264 1 application/vnd.ericsson.quickcall
265 1 application/vnd.eszigno3+xml
266 1 application/vnd.eudora.data
267 1 application/vnd.ezpix-album
268 1 application/vnd.ezpix-package
269 1 application/vnd.f-secure.mobile
270 1 application/vnd.fdf
271 1 application/vnd.fdsn.mseed
272 1 application/vnd.ffsns
273 1 application/vnd.fints
274 1 application/vnd.FloGraphIt
275 1 application/vnd.fluxtime.clip
276 1 application/vnd.font-fontforge-sfd
277 1 application/vnd.framemaker
278 1 application/vnd.frogans.fnc
279 1 application/vnd.frogans.ltf
280 1 application/vnd.fsc.weblaunch
281 1 application/vnd.fujitsu.oasys
282 1 application/vnd.fujitsu.oasys2
283 1 application/vnd.fujitsu.oasys3
284 1 application/vnd.fujitsu.oasysgp
285 1 application/vnd.fujitsu.oasysprs
286 1 application/vnd.fujixerox.ART4
287 1 application/vnd.fujixerox.ART-EX
288 1 application/vnd.fujixerox.ddd
289 1 application/vnd.fujixerox.docuworks
290 1 application/vnd.fujixerox.docuworks.binder
291 1 application/vnd.fujixerox.HBPL
292 1 application/vnd.fut-misnet
293 1 application/vnd.fuzzysheet
294 1 application/vnd.genomatix.tuxedo
295 1 application/vnd.geogebra.file
296 1 application/vnd.gmx (OBSOLETE)
297 1 application/vnd.google-earth.kml+xml
298 1 application/vnd.google-earth.kmz
299 1 application/vnd.grafeq
300 1 application/vnd.gridmp
301 1 application/vnd.groove-account
302 1 application/vnd.groove-help
303 1 application/vnd.groove-identity-message
304 1 application/vnd.groove-injector
305 1 application/vnd.groove-tool-message
306 1 application/vnd.groove-tool-template
307 1 application/vnd.groove-vcard
308 1 application/vnd.HandHeld-Entertainment+xml
309 1 application/vnd.hbci
310 1 application/vnd.hcl-bireports
311 1 application/vnd.hhe.lesson-player
312 1 application/vnd.hp-HPGL
313 1 application/vnd.hp-hpid
314 1 application/vnd.hp-hps
315 1 application/vnd.hp-jlyt
316 1 application/vnd.hp-PCL
317 1 application/vnd.hp-PCLXL
318 1 application/vnd.httphone
319 1 application/vnd.hydrostatix.sof-data
321 1 application/vnd.hzn-3d-crossword
322 1 application/vnd.ibm.afplinedata
323 1 application/vnd.ibm.electronic-media
324 1 application/vnd.ibm.MiniPay
325 1 application/vnd.ibm.modcap
326 1 application/vnd.ibm.rights-management
327 1 application/vnd.ibm.secure-container
328 1 application/vnd.iccprofile
329 1 application/vnd.igloader
330 1 application/vnd.immervision-ivp
331 1 application/vnd.immervision-ivu
332 1 application/vnd.informedcontrol.rms+xml
333 1 application/vnd.informix-visionary
334 1 application/vnd.intercon.formnet
335 1 application/vnd.intertrust.digibox
336 1 application/vnd.intertrust.nncp
337 1 application/vnd.intu.qbo
338 1 application/vnd.intu.qfx
339 1 application/vnd.iptc.g2.conceptitem+xml
340 1 application/vnd.iptc.g2.knowledgeitem+xml
341 1 application/vnd.iptc.g2.newsitem+xml
342 1 application/vnd.iptc.g2.packageitem+xml
343 1 application/vnd.ipunplugged.rcprofile
344 1 application/vnd.irepository.package+xml
345 1 application/vnd.is-xpr
346 1 application/vnd.jam
347 1 application/vnd.japannet-directory-service
348 1 application/vnd.japannet-jpnstore-wakeup
349 1 application/vnd.japannet-payment-wakeup
350 1 application/vnd.japannet-registration
351 1 application/vnd.japannet-registration-wakeup
352 1 application/vnd.japannet-setstore-wakeup
353 1 application/vnd.japannet-verification
354 1 application/vnd.japannet-verification-wakeup
355 1 application/vnd.jcp.javame.midlet-rms
356 1 application/vnd.jisp
357 1 application/vnd.joost.joda-archive
358 1 application/vnd.kahootz
359 1 application/vnd.kde.karbon
360 1 application/vnd.kde.kchart
361 1 application/vnd.kde.kformula
362 1 application/vnd.kde.kivio
363 1 application/vnd.kde.kontour
364 1 application/vnd.kde.kpresenter
365 1 application/vnd.kde.kspread
366 1 application/vnd.kde.kword
367 1 application/vnd.kenameaapp
368 1 application/vnd.kidspiration
369 1 application/vnd.Kinar
370 1 application/vnd.koan
371 1 application/vnd.kodak-descriptor
372 1 application/vnd.liberty-request+xml
373 1 application/vnd.llamagraphics.life-balance.desktop
374 1 application/vnd.llamagraphics.life-balance.exchange+xml
375 1 application/vnd.lotus-1-2-3
376 1 application/vnd.lotus-approach
377 1 application/vnd.lotus-freelance
378 1 application/vnd.lotus-notes
379 1 application/vnd.lotus-organizer
380 1 application/vnd.lotus-screencam
381 1 application/vnd.lotus-wordpro
382 1 application/vnd.macports.portpkg
383 1 application/vnd.marlin.drm.actiontoken+xml
384 1 application/vnd.marlin.drm.conftoken+xml
385 1 application/vnd.marlin.drm.license+xml
386 1 application/vnd.marlin.drm.mdcf
387 1 application/vnd.mcd
388 1 application/vnd.medcalcdata
389 1 application/vnd.mediastation.cdkey
390 1 application/vnd.meridian-slingshot
391 1 application/vnd.MFER
392 1 application/vnd.mfmp
393 1 application/vnd.micrografx.flo
394 1 application/vnd.micrografx.igx
397 1 application/vnd.mif
398 1 application/vnd.minisoft-hp3000-save
399 1 application/vnd.mitsubishi.misty-guard.trustweb
400 1 application/vnd.Mobius.DAF
401 1 application/vnd.Mobius.DIS
402 1 application/vnd.Mobius.MBK
403 1 application/vnd.Mobius.MQY
404 1 application/vnd.Mobius.MSL
405 1 application/vnd.Mobius.PLC
406 1 application/vnd.Mobius.TXF
407 1 application/vnd.mophun.application
408 1 application/vnd.mophun.certificate
409 1 application/vnd.motorola.flexsuite
410 1 application/vnd.motorola.flexsuite.adsi
411 1 application/vnd.motorola.flexsuite.fis
412 1 application/vnd.motorola.flexsuite.gotap
413 1 application/vnd.motorola.flexsuite.kmr
414 1 application/vnd.motorola.flexsuite.ttc
415 1 application/vnd.motorola.flexsuite.wem
416 1 application/vnd.motorola.iprm
417 1 application/vnd.mozilla.xul+xml
418 1 application/vnd.ms-artgalry
419 1 application/vnd.ms-asf
420 1 application/vnd.ms-cab-compressed
421 1 application/vnd.mseq
422 1 application/vnd.ms-excel
423 1 application/vnd.ms-fontobject
424 1 application/vnd.ms-htmlhelp
425 1 application/vnd.msign
426 1 application/vnd.ms-ims
427 1 application/vnd.ms-lrm
428 1 application/vnd.ms-playready.initiator+xml
429 1 application/vnd.ms-powerpoint
430 1 application/vnd.ms-project
431 1 application/vnd.ms-tnef
432 1 application/vnd.ms-wmdrm.lic-chlg-req
433 1 application/vnd.ms-wmdrm.lic-resp
434 1 application/vnd.ms-wmdrm.meter-chlg-req
435 1 application/vnd.ms-wmdrm.meter-resp
436 1 application/vnd.ms-works
437 1 application/vnd.ms-wpl
438 1 application/vnd.ms-xpsdocument
439 1 application/vnd.multiad.creator
440 1 application/vnd.multiad.creator.cif
441 1 application/vnd.musician
442 1 application/vnd.music-niff
443 1 application/vnd.muvee.style
444 1 application/vnd.ncd.control
445 1 application/vnd.ncd.reference
446 1 application/vnd.nervana
447 1 application/vnd.netfpx
448 1 application/vnd.neurolanguage.nlu
449 1 application/vnd.noblenet-directory
450 1 application/vnd.noblenet-sealer
451 1 application/vnd.noblenet-web
452 1 application/vnd.nokia.catalogs
453 1 application/vnd.nokia.conml+wbxml
454 1 application/vnd.nokia.conml+xml
455 1 application/vnd.nokia.iptv.config+xml
456 1 application/vnd.nokia.iSDS-radio-presets
457 1 application/vnd.nokia.landmark+wbxml
458 1 application/vnd.nokia.landmark+xml
459 1 application/vnd.nokia.landmarkcollection+xml
460 1 application/vnd.nokia.ncd
461 1 application/vnd.nokia.n-gage.ac+xml
462 1 application/vnd.nokia.n-gage.data
463 1 application/vnd.nokia.n-gage.symbian.install
464 1 application/vnd.nokia.pcd+wbxml
465 1 application/vnd.nokia.pcd+xml
466 1 application/vnd.nokia.radio-preset
467 1 application/vnd.nokia.radio-presets
468 1 application/vnd.novadigm.EDM
469 1 application/vnd.novadigm.EDX
470 1 application/vnd.novadigm.EXT
471 1 application/vnd.oasis.opendocument.chart
472 1 application/vnd.oasis.opendocument.chart-template
473 1 application/vnd.oasis.opendocument.database
474 1 application/vnd.oasis.opendocument.formula
475 1 application/vnd.oasis.opendocument.formula-template
476 1 application/vnd.oasis.opendocument.graphics
477 1 application/vnd.oasis.opendocument.graphics-template
478 1 application/vnd.oasis.opendocument.image
479 1 application/vnd.oasis.opendocument.image-template
480 1 application/vnd.oasis.opendocument.presentation
481 1 application/vnd.oasis.opendocument.presentation-template
482 1 application/vnd.oasis.opendocument.spreadsheet
483 1 application/vnd.oasis.opendocument.spreadsheet-template
484 1 application/vnd.oasis.opendocument.text
485 1 application/vnd.oasis.opendocument.text-master
486 1 application/vnd.oasis.opendocument.text-template
487 1 application/vnd.oasis.opendocument.text-web
488 1 application/vnd.obn
489 1 application/vnd.olpc-sugar
490 1 application/vnd.oma.bcast.associated-procedure-parameter+xml
491 1 application/vnd.oma.bcast.drm-trigger+xml
492 1 application/vnd.oma.bcast.imd+xml
493 1 application/vnd.oma.bcast.ltkm
494 1 application/vnd.oma.bcast.notification+xml
495 1 application/vnd.oma.bcast.provisioningtrigger
496 1 application/vnd.oma.bcast.sgboot
497 1 application/vnd.oma.bcast.sgdd+xml
498 1 application/vnd.oma.bcast.sgdu
499 1 application/vnd.oma.bcast.simple-symbol-container
500 1 application/vnd.oma.bcast.smartcard-trigger+xml
501 1 application/vnd.oma.bcast.sprov+xml
502 1 application/vnd.oma.bcast.stkm
503 1 application/vnd.oma.dcd
504 1 application/vnd.oma.dcdc
505 1 application/vnd.oma.dd2+xml
506 1 application/vnd.oma.drm.risd+xml
507 1 application/vnd.oma.group-usage-list+xml
508 1 application/vnd.oma.poc.detailed-progress-report+xml
509 1 application/vnd.oma.poc.final-report+xml
510 1 application/vnd.oma.poc.groups+xml
511 1 application/vnd.oma.poc.invocation-descriptor+xml
512 1 application/vnd.oma.poc.optimized-progress-report+xml
513 1 application/vnd.oma.xcap-directory+xml
514 1 application/vnd.omads-email+xml
515 1 application/vnd.omads-file+xml
516 1 application/vnd.omads-folder+xml
517 1 application/vnd.omaloc-supl-init
518 1 application/vnd.oma-scws-config
519 1 application/vnd.oma-scws-http-request
520 1 application/vnd.oma-scws-http-response
521 1 application/vnd.openofficeorg.extension
522 1 application/vnd.osa.netdeploy
523 1 application/vnd.osgi.bundle
524 1 application/vnd.osgi.dp
525 1 application/vnd.otps.ct-kip+xml
526 1 application/vnd.palm
527 1 application/vnd.paos.xml
528 1 application/vnd.pg.format
529 1 application/vnd.pg.osasli
530 1 application/vnd.piaccess.application-licence
531 1 application/vnd.picsel
532 1 application/vnd.poc.group-advertisement+xml
533 1 application/vnd.pocketlearn
534 1 application/vnd.powerbuilder6
535 1 application/vnd.powerbuilder6-s
536 1 application/vnd.powerbuilder7
537 1 application/vnd.powerbuilder75
538 1 application/vnd.powerbuilder75-s
539 1 application/vnd.powerbuilder7-s
540 1 application/vnd.preminet
541 1 application/vnd.previewsystems.box
542 1 application/vnd.proteus.magazine
543 1 application/vnd.publishare-delta-tree
544 1 application/vnd.pvi.ptid1
545 1 application/vnd.pwg-multiplexed
546 1 application/vnd.pwg-xhtml-print+xml
547 1 application/vnd.qualcomm.brew-app-res
548 1 application/vnd.Quark.QuarkXPress
549 1 application/vnd.rapid
550 1 application/vnd.recordare.musicxml
551 1 application/vnd.recordare.musicxml+xml
552 1 application/vnd.RenLearn.rlprint
553 1 application/vnd.route66.link66+xml
554 1 application/vnd.ruckus.download
555 1 application/vnd.s3sms
556 1 application/vnd.sbm.cid
557 1 application/vnd.sbm.mid2
558 1 application/vnd.scribus
559 1 application/vnd.sealed.3df
560 1 application/vnd.sealed.csf
561 1 application/vnd.sealed.doc
562 1 application/vnd.sealed.eml
563 1 application/vnd.sealed.mht
564 1 application/vnd.sealed.net
565 1 application/vnd.sealed.ppt
566 1 application/vnd.sealed.tiff
567 1 application/vnd.sealed.xls
568 1 application/vnd.sealedmedia.softseal.html
569 1 application/vnd.sealedmedia.softseal.pdf
570 1 application/vnd.seemail
571 1 application/vnd.sema
572 1 application/vnd.semd
573 1 application/vnd.semf
574 1 application/vnd.shana.informed.formdata
575 1 application/vnd.shana.informed.formtemplate
576 1 application/vnd.shana.informed.interchange
577 1 application/vnd.shana.informed.package
578 1 application/vnd.SimTech-MindMapper
579 1 application/vnd.smaf
580 1 application/vnd.smart.teacher
581 1 application/vnd.software602.filler.form+xml
582 1 application/vnd.software602.filler.form-xml-zip
583 1 application/vnd.solent.sdkm+xml
584 1 application/vnd.spotfire.dxp
585 1 application/vnd.spotfire.sfs
586 1 application/vnd.sss-cod
587 1 application/vnd.sss-dtf
588 1 application/vnd.sss-ntf
589 1 application/vnd.street-stream
590 1 application/vnd.sun.wadl+xml
591 1 application/vnd.sus-calendar
592 1 application/vnd.svd
593 1 application/vnd.swiftview-ics
594 1 application/vnd.syncml.dm+wbxml
595 1 application/vnd.syncml.dm+xml
596 1 application/vnd.syncml.ds.notification
597 1 application/vnd.syncml+xml
598 1 application/vnd.tao.intent-module-archive
599 1 application/vnd.tmobile-livetv
600 1 application/vnd.trid.tpt
601 1 application/vnd.triscape.mxs
602 1 application/vnd.trueapp
603 1 application/vnd.truedoc
604 1 application/vnd.ufdl
605 1 application/vnd.uiq.theme
606 1 application/vnd.umajin
607 1 application/vnd.unity
608 1 application/vnd.uoml+xml
609 1 application/vnd.uplanet.alert
610 1 application/vnd.uplanet.alert-wbxml
611 1 application/vnd.uplanet.bearer-choice
612 1 application/vnd.uplanet.bearer-choice-wbxml
613 1 application/vnd.uplanet.cacheop
614 1 application/vnd.uplanet.cacheop-wbxml
615 1 application/vnd.uplanet.channel
616 1 application/vnd.uplanet.channel-wbxml
617 1 application/vnd.uplanet.list
618 1 application/vnd.uplanet.listcmd
619 1 application/vnd.uplanet.listcmd-wbxml
620 1 application/vnd.uplanet.list-wbxml
621 1 application/vnd.uplanet.signal
622 1 application/vnd.vcx
623 1 application/vnd.vd-study
624 1 application/vnd.vectorworks
625 1 application/vnd.vidsoft.vidconference
626 1 application/vnd.visio
627 1 application/vnd.visionary
628 1 application/vnd.vividence.scriptfile
629 1 application/vnd.vsf
630 1 application/vnd.wap.sic
631 1 application/vnd.wap.slc
632 1 application/vnd.wap.wbxml
633 1 application/vnd.wap.wmlc
634 1 application/vnd.wap.wmlscriptc
635 1 application/vnd.webturbo
636 1 application/vnd.wfa.wsc
637 1 application/vnd.wmc
638 1 application/vnd.wmf.bootstrap
639 1 application/vnd.wordperfect
640 1 application/vnd.wqd
641 1 application/vnd.wrq-hp3000-labelled
642 1 application/vnd.wt.stf
643 1 application/vnd.wv.csp+xml
644 1 application/vnd.wv.csp+wbxml
645 1 application/vnd.wv.ssp+xml
646 1 application/vnd.xara
647 1 application/vnd.xfdl
648 1 application/vnd.xfdl.webform
649 1 application/vnd.xmi+xml
650 1 application/vnd.xmpie.cpkg
651 1 application/vnd.xmpie.dpkg
652 1 application/vnd.xmpie.plan
653 1 application/vnd.xmpie.ppkg
654 1 application/vnd.xmpie.xlim
655 1 application/vnd.yamaha.hv-dic
656 1 application/vnd.yamaha.hv-script
657 1 application/vnd.yamaha.hv-voice
658 1 application/vnd.yamaha.openscoreformat
659 1 application/vnd.yamaha.smaf-audio
660 1 application/vnd.yamaha.smaf-phrase
661 1 application/vnd.yellowriver-custom-menu
662 1 application/vnd.zul
663 1 application/vnd.zzazz.deck+xml
664 1 application/voicexml+xml
665 1 application/watcherinfo+xml
666 1 application/whoispp-query
667 1 application/whoispp-response
668 1 application/wita
669 1 application/wordperfect5.1
670 1 application/wsdl+xml
671 1 application/wspolicy+xml
672 1 application/x400-bp
673 1 application/xcap-att+xml
674 1 application/xcap-caps+xml
675 1 application/xcap-el+xml
676 1 application/xcap-error+xml
677 1 application/xcap-ns+xml
678 1 application/xcon-conference-info-diff+xml
679 1 application/xcon-conference-info+xml
680 1 application/xenc+xml
681 1 application/xhtml-voice+xml (Obsolete)
682 1 application/xhtml+xml
683 1 application/xml
684 1 application/xml-dtd
685 1 application/xml-external-parsed-entity
686 1 application/xmpp+xml
687 1 application/xop+xml
688 1 application/xv+xml
689 1 application/zip
690 2 audio/32kadpcm
691 2 audio/3gpp
692 2 audio/3gpp2
693 2 audio/ac3
694 2 audio/AMR
695 2 audio/AMR-WB
696 2 audio/amr-wb+
697 2 audio/asc
698 2 audio/basic
699 2 audio/BV16
700 2 audio/BV32
701 2 audio/clearmode
702 2 audio/CN
703 2 audio/DAT12
704 2 audio/dls
705 2 audio/dsr-es201108
706 2 audio/dsr-es202050
707 2 audio/dsr-es202211
708 2 audio/dsr-es202212
709 2 audio/eac3
710 2 audio/DVI4
711 2 audio/EVRC
712 2 audio/EVRC0
713 2 audio/EVRC1
714 2 audio/EVRCB
715 2 audio/EVRCB0
716 2 audio/EVRCB1
717 2 audio/EVRC-QCP
718 2 audio/EVRCWB
719 2 audio/EVRCWB0
720 2 audio/EVRCWB1
721 2 audio/example
722 2 audio/G719
723 2 audio/G722
724 2 audio/G7221
725 2 audio/G723
726 2 audio/G726-16
727 2 audio/G726-24
728 2 audio/G726-32
729 2 audio/G726-40
730 2 audio/G728
731 2 audio/G729
732 2 audio/G7291
733 2 audio/G729D
734 2 audio/G729E
735 2 audio/GSM
736 2 audio/GSM-EFR
737 2 audio/iLBC
738 2 audio/L8
739 2 audio/L16
740 2 audio/L20
741 2 audio/L24
742 2 audio/LPC
743 2 audio/mobile-xmf
744 2 audio/MPA
745 2 audio/mp4
746 2 audio/MP4A-LATM
747 2 audio/mpa-robust
748 2 audio/mpeg
749 2 audio/mpeg4-generic
750 2 audio/ogg
751 2 audio/parityfec
752 2 audio/PCMA
753 2 audio/PCMA-WB
754 2 audio/PCMU
755 2 audio/PCMU-WB
756 2 audio/prs.sid
757 2 audio/QCELP
758 2 audio/RED
759 2 audio/rtp-enc-aescm128
760 2 audio/rtp-midi
761 2 audio/rtx
762 2 audio/SMV
763 2 audio/SMV0
764 2 audio/SMV-QCP
765 2 audio/sp-midi
766 2 audio/t140c
767 2 audio/t38
768 2 audio/telephone-event
769 2 audio/tone
770 2 audio/ulpfec
771 2 audio/VDVI
772 2 audio/VMR-WB
773 2 audio/vnd.3gpp.iufp
774 2 audio/vnd.4SB
775 2 audio/vnd.audiokoz
776 2 audio/vnd.CELP
777 2 audio/vnd.cisco.nse
778 2 audio/vnd.cmles.radio-events
779 2 audio/vnd.cns.anp1
780 2 audio/vnd.cns.inf1
781 2 audio/vnd.digital-winds
782 2 audio/vnd.dlna.adts
783 2 audio/vnd.dolby.heaac.1
784 2 audio/vnd.dolby.mlp
785 2 audio/vnd.dolby.mps
786 2 audio/vnd.dts
787 2 audio/vnd.dts.hd
788 2 audio/vnd.everad.plj
789 2 audio/vnd.hns.audio
790 2 audio/vnd.lucent.voice
791 2 audio/vnd.ms-playready.media.pya
792 2 audio/vnd.nokia.mobile-xmf
793 2 audio/vnd.nortel.vbk
794 2 audio/vnd.nuera.ecelp4800
795 2 audio/vnd.nuera.ecelp7470
796 2 audio/vnd.nuera.ecelp9600
797 2 audio/vnd.octel.sbc
798 2 audio/vnd.qcelp - DEPRECATED - Please use audio/qcelp
799 2 audio/vnd.rhetorex.32kadpcm
800 2 audio/vnd.sealedmedia.softseal.mpeg
801 2 audio/vnd.vmx.cvsd
802 2 audio/vorbis
803 2 audio/vorbis-config
804 4 image/cgm
805 4 image/example
806 4 image/fits
807 4 image/g3fax
808 4 image/gif
809 4 image/ief
810 4 image/jp2
811 4 image/jpeg
812 4 image/jpm
813 4 image/jpx
814 4 image/naplps
815 4 image/png
816 4 image/prs.btif
817 4 image/prs.pti
818 4 image/t38
819 4 image/tiff
820 4 image/tiff-fx
821 4 image/vnd.adobe.photoshop
822 4 image/vnd.cns.inf2
823 4 image/vnd.djvu
824 4 image/vnd.dwg
825 4 image/vnd.dxf
826 4 image/vnd.fastbidsheet
827 4 image/vnd.fpx
828 4 image/vnd.fst
829 4 image/vnd.fujixerox.edmics-mmr
830 4 image/vnd.fujixerox.edmics-rlc
831 4 image/vnd.globalgraphics.pgb
832 4 image/vnd.microsoft.icon
833 4 image/vnd.mix
834 4 image/vnd.ms-modi
835 4 image/vnd.net-fpx
836 4 image/vnd.radiance
837 4 image/vnd.sealed.png
838 4 image/vnd.sealedmedia.softseal.gif
839 4 image/vnd.sealedmedia.softseal.jpg
840 4 image/vnd.svf
841 4 image/vnd.wap.wbmp
842 4 image/vnd.xiff
843 5 message/CPIM
844 5 message/delivery-status
845 5 message/disposition-notification
846 5 message/example
847 5 message/external-body
848 5 message/global
849 5 message/global-delivery-status
850 5 message/global-disposition-notification
851 5 message/global-headers
852 5 message/http
853 5 message/imdn+xml
854 5 message/news
855 5 message/partial
856 5 message/rfc822
857 5 message/s-http
858 5 message/sip
859 5 message/sipfrag
860 5 message/tracking-status
861 5 message/vnd.si.simp
862 6 model/example
863 6 model/iges
864 6 model/mesh
865 6 model/vnd.dwf
866 6 model/vnd.flatland.3dml
867 6 model/vnd.gdl
868 6 model/vnd.gs-gdl
869 6 model/vnd.gtw
870 6 model/vnd.moml+xml
871 6 model/vnd.mts
872 6 model/vnd.parasolid.transmit.binary
873 6 model/vnd.parasolid.transmit.text
874 6 model/vnd.vtu
875 6 model/vrml
876 7 multipart/alternative
877 7 multipart/appledouble
878 7 multipart/byteranges
879 7 multipart/digest
880 7 multipart/encrypted
881 7 multipart/example
882 7 multipart/form-data
883 7 multipart/header-set
884 7 multipart/mixed
885 7 multipart/parallel
886 7 multipart/related
887 7 multipart/report
888 7 multipart/signed
889 7 multipart/voice-message
890 8 text/calendar
891 8 text/css
892 8 text/csv
893 8 text/directory
894 8 text/dns
895 8 text/ecmascript (obsolete)
896 8 text/enriched
897 8 text/example
898 8 text/html
899 8 text/javascript (obsolete)
900 8 text/parityfec
901 8 text/plain
902 8 text/prs.fallenstein.rst
903 8 text/prs.lines.tag
904 8 text/RED
905 8 text/rfc822-headers
906 8 text/richtext
907 8 text/rtf
908 8 text/rtp-enc-aescm128
909 8 text/rtx
910 8 text/sgml
911 8 text/t140
912 8 text/tab-separated-values
913 8 text/troff
914 8 text/ulpfec
915 8 text/uri-list
916 8 text/vnd.abc
917 8 text/vnd.curl
918 8 text/vnd.DMClientScript
919 8 text/vnd.esmertec.theme-descriptor
920 8 text/vnd.fly
921 8 text/vnd.fmi.flexstor
922 8 text/vnd.graphviz
923 8 text/vnd.in3d.3dml
924 8 text/vnd.in3d.spot
925 8 text/vnd.IPTC.NewsML
926 8 text/vnd.IPTC.NITF
927 8 text/vnd.latex-z
928 8 text/vnd.motorola.reflex
929 8 text/vnd.ms-mediapackage
930 8 text/vnd.net2phone.commcenter.command
931 8 text/vnd.si.uricatalogue
932 8 text/vnd.sun.j2me.app-descriptor
933 8 text/vnd.trolltech.linguist
934 8 text/vnd.wap.si
935 8 text/vnd.wap.sl
936 8 text/vnd.wap.wml
937 8 text/vnd.wap.wmlscript
938 8 text/xml
939 8 text/xml-external-parsed-entity
940 9 video/3gpp
941 9 video/3gpp2
942 9 video/3gpp-tt
943 9 video/BMPEG
944 9 video/BT656
945 9 video/CelB
946 9 video/DV
947 9 video/example
948 9 video/H261
949 9 video/H263
950 9 video/H263-1998
951 9 video/H263-2000
952 9 video/H264
953 9 video/JPEG
954 9 video/jpeg2000
955 9 video/MJ2
956 9 video/MP1S
957 9 video/MP2P
958 9 video/MP2T
959 9 video/mp4
960 9 video/MP4V-ES
961 9 video/MPV
962 9 video/mpeg
963 9 video/mpeg4-generic
964 9 video/nv
965 9 video/ogg
966 9 video/parityfec
967 9 video/pointer
968 9 video/quicktime
969 9 video/raw
970 9 video/rtp-enc-aescm128
971 9 video/rtx
972 9 video/SMPTE292M
973 9 video/ulpfec
974 9 video/vc1
975 9 video/vnd.CCTV
976 9 video/vnd.dlna.mpeg-tts
977 9 video/vnd.fvt
978 9 video/vnd.hns.video
979 9 video/vnd.iptvforum.1dparityfec-1010
980 9 video/vnd.iptvforum.1dparityfec-2005
981 9 video/vnd.iptvforum.2dparityfec-1010
982 9 video/vnd.iptvforum.2dparityfec-2005
983 9 video/vnd.iptvforum.ttsavc
984 9 video/vnd.iptvforum.ttsmpeg2
985 9 video/vnd.motorola.video
986 9 video/vnd.motorola.videop
987 9 video/vnd.mpegurl
988 9 video/vnd.ms-playready.media.pyv
989 9 video/vnd.nokia.interleaved-multimedia
990 9 video/vnd.nokia.videovoip
991 9 video/vnd.objectvideo
992 9 video/vnd.sealed.mpeg1
993 9 video/vnd.sealed.mpeg4
994 9 video/vnd.sealed.swf
995 9 video/vnd.sealedmedia.softseal.mov
996 9 video/vnd.vivo
997 6 image/x-icon
998 4 image/pjpeg
999 4 image/x-xbitmap
1000 1 application/x-shockwave-flash
1001 4 image/svg+xml
1002 4 image/jpg

Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
1,905 Posts

Another condition that can occur that throw a 406 is when you ask for file such as a GIF:

But we actually get a JAVASCRIPT back, which be default, isn't allowed.

The reason for the bait and switch could be tracking, or to save bandwidth for relatively unfamiliar crawlers, or due to a coding error.

With a bit of code to check for another AbsoluteUri responding that differs from the WebClient.RequestUri, we don't see the 406 problem.

For a reason that I currently don't know, .gifs are more prone to redirection, and in the default configuration of arachnode.net, to an unsupported AllowedDataType.

-Mike

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
1,905 Posts

I have commented a line in DataManager.cs (to be checked in) that sets the Accept property of the WebClient.  I need to do some additional digging to find out why .gifs and .css files are often rejected.

However, since AN will now make HEAD requests before downloading the content stream, we can check for AllowedDataTypes.  More to come...

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Top 10 Contributor
229 Posts

this is exactly who this guy solved it out:

developer.yahoo.net/forum/index.php?showtopic=76

Top 10 Contributor
1,905 Posts

Thanks.

For best service when you require assistance:

  1. Check the DisallowedAbsoluteUris and Exceptions tables first.
  2. Cut and paste actual exceptions from the Exceptions table.
  3. Include screenshots.

Skype: arachnodedotnet

Page 1 of 1 (6 items) | RSS
An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

copyright 2004-2017, arachnode.net LLC