I am gettting 406 exception when try to get an image. why is that?
AbsoluteUri1http://10net.co.il/108752/%D7%A6%D7%A4%D7%99%D7%99%D7%94-%D7%99%D7%A9%D7%99%D7%A8%D7%94-%D7%91%D7%A1%D7%A8%D7%98%D7%99-%D7%93%D7%A8%D7%9E%D7%94
AbsoluteUri2http://10net.co.il/image/users/108752/ftp/my_files/%D7%A1%D7%A8%D7%98-%D7%93%D7%A8%D7%9E%D7%94.jpg
MessageThe remote server returned an error: (406) Not Acceptable.
SourceSystem
StackTrace at System.Net.HttpWebRequest.GetResponse() at System.Net.WebClient.GetWebResponse(WebRequest request) at Arachnode.SiteCrawler.Components.WebClient.GetWebResponse(WebRequest request)
This is due to certain WebServers not handling the HttpRequest header 'Accept-Types' properly.
You really are going to make me fix all of my little things, huh?
I'll take a look...
Just for a little background on why you may get this error. Arachnode.net uses a very specific set of configurables to determine what to download, so you know exactly what what you are downloading. A full set of ContentTypes is provided in cfg.ContentTypes.
In order to download a piece of content you will need to make an entry in cfg.AllowedDataTypes. The most common internet DataTypes are already present in the default configuration.
There are some notable exceptions: Movies (huge), and JavaScripts (possibly malicious)..., and other scripts (also, possibly malicious)...
The FullTextIndexType column tells SQL and the iFilters (if installed) how to treat the source byte array when using SQL Full Text Indexing, if enabled. Overrides allow the FullTextIndexType column to be more specific, should your iFilters require additional specificity. (that was slightly redundant...!)
Here is where the 'Accept' HttpRequest header is assigned (WebClient.cs):
A list of all arachnode.net supplied Content-Types:
1 0 UNKNOWN2 1 application/activemessage3 1 application/andrew-inset4 1 application/applefile5 1 application/atom+xml6 1 application/atomicmail7 1 application/atomcat+xml8 1 application/atomsvc+xml9 1 application/auth-policy+xml10 1 application/batch-SMTP11 1 application/beep+xml12 1 application/cals-184013 1 application/ccxml+xml14 1 application/cea-2018+xml15 1 application/cellml+xml16 1 application/cnrp+xml17 1 application/commonground18 1 application/conference-info+xml19 1 application/cpl+xml20 1 application/csta+xml21 1 application/CSTAdata+xml22 1 application/cybercash23 1 application/davmount+xml24 1 application/dca-rft25 1 application/dec-dx26 1 application/dialog-info+xml27 1 application/dicom28 1 application/dns29 1 application/dvcs30 1 application/ecmascript31 1 application/EDI-Consent32 1 application/EDIFACT33 1 application/EDI-X1234 1 application/emma+xml35 1 application/epp+xml36 1 application/eshop37 1 application/example38 1 application/fastinfoset39 1 application/fastsoap40 1 application/fits41 1 application/font-tdpfr42 1 application/H22443 1 application/http44 1 application/hyperstudio45 1 application/ibe-key-request+xml46 1 application/ibe-pkg-reply+xml47 1 application/ibe-pp-data48 1 application/iges49 1 application/im-iscomposing+xml50 1 application/index51 1 application/index.cmd52 1 application/index.obj53 1 application/index.response54 1 application/index.vnd55 1 application/iotp56 1 application/ipp57 1 application/isup58 1 application/javascript59 1 application/json60 1 application/kpml-request+xml61 1 application/kpml-response+xml62 1 application/lost+xml63 1 application/mac-binhex4064 1 application/macwriteii65 1 application/marc66 1 application/mathematica67 1 application/mbms-associated-procedure-description+xml68 1 application/mbms-deregister+xml69 1 application/mbms-envelope+xml70 1 application/mbms-msk-response+xml71 1 application/mbms-msk+xml72 1 application/mbms-protection-description+xml73 1 application/mbms-reception-report+xml74 1 application/mbms-register-response+xml75 1 application/mbms-register+xml76 1 application/mbms-user-service-description+xml77 1 application/mbox78 1 application/media_control+xml79 1 application/mediaservercontrol+xml80 1 application/mikey81 1 application/moss-keys82 1 application/moss-signature83 1 application/mosskey-data84 1 application/mosskey-request85 1 application/mpeg4-generic86 1 application/mpeg4-iod87 1 application/mpeg4-iod-xmt88 1 application/mp489 1 application/msword90 1 application/mxf91 1 application/nasdata92 1 application/news-transmission93 1 application/nss94 1 application/ocsp-request95 1 application/ocsp-response96 1 application/octet-stream97 1 application/oda98 1 application/oebps-package+xml99 1 application/ogg100 1 application/parityfec101 1 application/patch-ops-error+xml102 1 application/pdf103 1 application/pgp-encrypted104 1 application/pgp-keys105 1 application/pgp-signature106 1 application/pidf+xml107 1 application/pidf-diff+xml108 1 application/pkcs10109 1 application/pkcs7-mime110 1 application/pkcs7-signature111 1 application/pkix-cert112 1 application/pkixcmp113 1 application/pkix-crl114 1 application/pkix-pkipath115 1 application/pls+xml116 1 application/poc-settings+xml117 1 application/postscript118 1 application/prs.alvestrand.titrax-sheet119 1 application/prs.cww120 1 application/prs.nprend121 1 application/prs.plucker122 1 application/rdf+xml123 1 application/qsig124 1 application/reginfo+xml125 1 application/relax-ng-compact-syntax126 1 application/remote-printing127 1 application/resource-lists-diff+xml128 1 application/resource-lists+xml129 1 application/riscos130 1 application/rlmi+xml131 1 application/rls-services+xml132 1 application/rtf133 1 application/rtx134 1 application/samlassertion+xml135 1 application/samlmetadata+xml136 1 application/sbml+xml137 1 application/scvp-cv-request138 1 application/scvp-cv-response139 1 application/scvp-vp-request140 1 application/scvp-vp-response141 1 application/sdp142 1 application/set-payment143 1 application/set-payment-initiation144 1 application/set-registration145 1 application/set-registration-initiation146 1 application/sgml147 1 application/sgml-open-catalog148 1 application/shf+xml149 1 application/sieve150 1 application/simple-filter+xml151 1 application/simple-message-summary152 1 application/simpleSymbolContainer153 1 application/slate154 1 application/smil (OBSOLETE)155 1 application/smil+xml156 1 application/soap+fastinfoset157 1 application/soap+xml158 1 application/sparql-query159 1 application/sparql-results+xml160 1 application/spirits-event+xml161 1 application/srgs162 1 application/srgs+xml163 1 application/ssml+xml164 1 application/timestamp-query165 1 application/timestamp-reply166 1 application/tve-trigger167 1 application/ulpfec168 1 application/vemmi169 1 application/vnd.3gpp.bsf+xml170 1 application/vnd.3gpp.pic-bw-large171 1 application/vnd.3gpp.pic-bw-small172 1 application/vnd.3gpp.pic-bw-var173 1 application/vnd.3gpp.sms174 1 application/vnd.3gpp2.bcmcsinfo+xml175 1 application/vnd.3gpp2.sms176 1 application/vnd.3gpp2.tcap177 1 application/vnd.3M.Post-it-Notes178 1 application/vnd.accpac.simply.aso179 1 application/vnd.accpac.simply.imp180 1 application/vnd.acucobol181 1 application/vnd.acucorp182 1 application/vnd.adobe.xdp+xml183 1 application/vnd.adobe.xfdf184 1 application/vnd.aether.imp185 1 application/vnd.airzip.filesecure.azf186 1 application/vnd.airzip.filesecure.azs187 1 application/vnd.americandynamics.acc188 1 application/vnd.amiga.ami189 1 application/vnd.anser-web-certificate-issue-initiation190 1 application/vnd.antix.game-component191 1 application/vnd.apple.installer+xml192 1 application/vnd.arastra.swi193 1 application/vnd.audiograph194 1 application/vnd.autopackage195 1 application/vnd.avistar+xml196 1 application/vnd.blueice.multipass197 1 application/vnd.bluetooth.ep.oob198 1 application/vnd.bmi199 1 application/vnd.businessobjects200 1 application/vnd.cab-jscript201 1 application/vnd.canon-cpdl202 1 application/vnd.canon-lips203 1 application/vnd.cendio.thinlinc.clientconf204 1 application/vnd.chemdraw+xml205 1 application/vnd.chipnuts.karaoke-mmd206 1 application/vnd.cinderella207 1 application/vnd.cirpack.isdn-ext208 1 application/vnd.claymore209 1 application/vnd.clonk.c4group210 1 application/vnd.commerce-battelle211 1 application/vnd.commonspace212 1 application/vnd.cosmocaller213 1 application/vnd.contact.cmsg214 1 application/vnd.crick.clicker215 1 application/vnd.crick.clicker.keyboard216 1 application/vnd.crick.clicker.palette217 1 application/vnd.crick.clicker.template218 1 application/vnd.crick.clicker.wordbank219 1 application/vnd.criticaltools.wbs+xml220 1 application/vnd.ctc-posml221 1 application/vnd.ctct.ws+xml222 1 application/vnd.cups-pdf223 1 application/vnd.cups-postscript224 1 application/vnd.cups-ppd225 1 application/vnd.cups-raster226 1 application/vnd.cups-raw227 1 application/vnd.curl228 1 application/vnd.cybank229 1 application/vnd.data-vision.rdz230 1 application/vnd.denovo.fcselayout-link231 1 application/vnd.dir-bi.plate-dl-nosuffix232 1 application/vnd.dna233 1 application/vnd.dpgraph234 1 application/vnd.dreamfactory235 1 application/vnd.dvb.esgcontainer236 1 application/vnd.dvb.ipdcdftnotifaccess237 1 application/vnd.dvb.ipdcesgaccess238 1 application/vnd.dvb.iptv.alfec-base240 1 application/vnd.dvb.iptv.alfec-enhancement241 1 application/vnd.dvb.notif-aggregate-root+xml242 1 application/vnd.dvb.notif-container+xml243 1 application/vnd.dvb.notif-generic+xml244 1 application/vnd.dvb.notif-ia-msglist+xml245 1 application/vnd.dvb.notif-ia-registration-request+xml246 1 application/vnd.dvb.notif-ia-registration-response+xml247 1 application/vnd.dvb.notif-init+xml248 1 application/vnd.dxr249 1 application/vnd.dynageo250 1 application/vnd.ecdis-update251 1 application/vnd.ecowin.chart252 1 application/vnd.ecowin.filerequest253 1 application/vnd.ecowin.fileupdate254 1 application/vnd.ecowin.series255 1 application/vnd.ecowin.seriesrequest256 1 application/vnd.ecowin.seriesupdate257 1 application/vnd.emclient.accessrequest+xml258 1 application/vnd.enliven259 1 application/vnd.epson.esf260 1 application/vnd.epson.msf261 1 application/vnd.epson.quickanime262 1 application/vnd.epson.salt263 1 application/vnd.epson.ssf264 1 application/vnd.ericsson.quickcall265 1 application/vnd.eszigno3+xml266 1 application/vnd.eudora.data267 1 application/vnd.ezpix-album268 1 application/vnd.ezpix-package269 1 application/vnd.f-secure.mobile270 1 application/vnd.fdf271 1 application/vnd.fdsn.mseed272 1 application/vnd.ffsns273 1 application/vnd.fints274 1 application/vnd.FloGraphIt275 1 application/vnd.fluxtime.clip276 1 application/vnd.font-fontforge-sfd277 1 application/vnd.framemaker278 1 application/vnd.frogans.fnc279 1 application/vnd.frogans.ltf280 1 application/vnd.fsc.weblaunch281 1 application/vnd.fujitsu.oasys282 1 application/vnd.fujitsu.oasys2283 1 application/vnd.fujitsu.oasys3284 1 application/vnd.fujitsu.oasysgp285 1 application/vnd.fujitsu.oasysprs286 1 application/vnd.fujixerox.ART4287 1 application/vnd.fujixerox.ART-EX288 1 application/vnd.fujixerox.ddd289 1 application/vnd.fujixerox.docuworks290 1 application/vnd.fujixerox.docuworks.binder291 1 application/vnd.fujixerox.HBPL292 1 application/vnd.fut-misnet293 1 application/vnd.fuzzysheet294 1 application/vnd.genomatix.tuxedo295 1 application/vnd.geogebra.file296 1 application/vnd.gmx (OBSOLETE)297 1 application/vnd.google-earth.kml+xml298 1 application/vnd.google-earth.kmz299 1 application/vnd.grafeq300 1 application/vnd.gridmp301 1 application/vnd.groove-account302 1 application/vnd.groove-help303 1 application/vnd.groove-identity-message304 1 application/vnd.groove-injector305 1 application/vnd.groove-tool-message306 1 application/vnd.groove-tool-template307 1 application/vnd.groove-vcard308 1 application/vnd.HandHeld-Entertainment+xml309 1 application/vnd.hbci310 1 application/vnd.hcl-bireports311 1 application/vnd.hhe.lesson-player312 1 application/vnd.hp-HPGL313 1 application/vnd.hp-hpid314 1 application/vnd.hp-hps315 1 application/vnd.hp-jlyt316 1 application/vnd.hp-PCL317 1 application/vnd.hp-PCLXL318 1 application/vnd.httphone319 1 application/vnd.hydrostatix.sof-data321 1 application/vnd.hzn-3d-crossword322 1 application/vnd.ibm.afplinedata323 1 application/vnd.ibm.electronic-media324 1 application/vnd.ibm.MiniPay325 1 application/vnd.ibm.modcap326 1 application/vnd.ibm.rights-management327 1 application/vnd.ibm.secure-container328 1 application/vnd.iccprofile329 1 application/vnd.igloader330 1 application/vnd.immervision-ivp331 1 application/vnd.immervision-ivu332 1 application/vnd.informedcontrol.rms+xml333 1 application/vnd.informix-visionary334 1 application/vnd.intercon.formnet335 1 application/vnd.intertrust.digibox336 1 application/vnd.intertrust.nncp337 1 application/vnd.intu.qbo338 1 application/vnd.intu.qfx339 1 application/vnd.iptc.g2.conceptitem+xml340 1 application/vnd.iptc.g2.knowledgeitem+xml341 1 application/vnd.iptc.g2.newsitem+xml342 1 application/vnd.iptc.g2.packageitem+xml343 1 application/vnd.ipunplugged.rcprofile344 1 application/vnd.irepository.package+xml345 1 application/vnd.is-xpr346 1 application/vnd.jam347 1 application/vnd.japannet-directory-service348 1 application/vnd.japannet-jpnstore-wakeup349 1 application/vnd.japannet-payment-wakeup350 1 application/vnd.japannet-registration351 1 application/vnd.japannet-registration-wakeup352 1 application/vnd.japannet-setstore-wakeup353 1 application/vnd.japannet-verification354 1 application/vnd.japannet-verification-wakeup355 1 application/vnd.jcp.javame.midlet-rms356 1 application/vnd.jisp357 1 application/vnd.joost.joda-archive358 1 application/vnd.kahootz359 1 application/vnd.kde.karbon360 1 application/vnd.kde.kchart361 1 application/vnd.kde.kformula362 1 application/vnd.kde.kivio363 1 application/vnd.kde.kontour364 1 application/vnd.kde.kpresenter365 1 application/vnd.kde.kspread366 1 application/vnd.kde.kword367 1 application/vnd.kenameaapp368 1 application/vnd.kidspiration369 1 application/vnd.Kinar370 1 application/vnd.koan371 1 application/vnd.kodak-descriptor372 1 application/vnd.liberty-request+xml373 1 application/vnd.llamagraphics.life-balance.desktop374 1 application/vnd.llamagraphics.life-balance.exchange+xml375 1 application/vnd.lotus-1-2-3376 1 application/vnd.lotus-approach377 1 application/vnd.lotus-freelance378 1 application/vnd.lotus-notes379 1 application/vnd.lotus-organizer380 1 application/vnd.lotus-screencam381 1 application/vnd.lotus-wordpro382 1 application/vnd.macports.portpkg383 1 application/vnd.marlin.drm.actiontoken+xml384 1 application/vnd.marlin.drm.conftoken+xml385 1 application/vnd.marlin.drm.license+xml386 1 application/vnd.marlin.drm.mdcf387 1 application/vnd.mcd388 1 application/vnd.medcalcdata389 1 application/vnd.mediastation.cdkey390 1 application/vnd.meridian-slingshot391 1 application/vnd.MFER392 1 application/vnd.mfmp393 1 application/vnd.micrografx.flo394 1 application/vnd.micrografx.igx397 1 application/vnd.mif398 1 application/vnd.minisoft-hp3000-save399 1 application/vnd.mitsubishi.misty-guard.trustweb400 1 application/vnd.Mobius.DAF401 1 application/vnd.Mobius.DIS402 1 application/vnd.Mobius.MBK403 1 application/vnd.Mobius.MQY404 1 application/vnd.Mobius.MSL405 1 application/vnd.Mobius.PLC406 1 application/vnd.Mobius.TXF407 1 application/vnd.mophun.application408 1 application/vnd.mophun.certificate409 1 application/vnd.motorola.flexsuite410 1 application/vnd.motorola.flexsuite.adsi411 1 application/vnd.motorola.flexsuite.fis412 1 application/vnd.motorola.flexsuite.gotap413 1 application/vnd.motorola.flexsuite.kmr414 1 application/vnd.motorola.flexsuite.ttc415 1 application/vnd.motorola.flexsuite.wem416 1 application/vnd.motorola.iprm417 1 application/vnd.mozilla.xul+xml418 1 application/vnd.ms-artgalry419 1 application/vnd.ms-asf420 1 application/vnd.ms-cab-compressed421 1 application/vnd.mseq422 1 application/vnd.ms-excel423 1 application/vnd.ms-fontobject424 1 application/vnd.ms-htmlhelp425 1 application/vnd.msign426 1 application/vnd.ms-ims427 1 application/vnd.ms-lrm428 1 application/vnd.ms-playready.initiator+xml429 1 application/vnd.ms-powerpoint430 1 application/vnd.ms-project431 1 application/vnd.ms-tnef432 1 application/vnd.ms-wmdrm.lic-chlg-req433 1 application/vnd.ms-wmdrm.lic-resp434 1 application/vnd.ms-wmdrm.meter-chlg-req435 1 application/vnd.ms-wmdrm.meter-resp436 1 application/vnd.ms-works437 1 application/vnd.ms-wpl438 1 application/vnd.ms-xpsdocument439 1 application/vnd.multiad.creator440 1 application/vnd.multiad.creator.cif441 1 application/vnd.musician442 1 application/vnd.music-niff443 1 application/vnd.muvee.style444 1 application/vnd.ncd.control445 1 application/vnd.ncd.reference446 1 application/vnd.nervana447 1 application/vnd.netfpx448 1 application/vnd.neurolanguage.nlu449 1 application/vnd.noblenet-directory450 1 application/vnd.noblenet-sealer451 1 application/vnd.noblenet-web452 1 application/vnd.nokia.catalogs453 1 application/vnd.nokia.conml+wbxml454 1 application/vnd.nokia.conml+xml455 1 application/vnd.nokia.iptv.config+xml456 1 application/vnd.nokia.iSDS-radio-presets457 1 application/vnd.nokia.landmark+wbxml458 1 application/vnd.nokia.landmark+xml459 1 application/vnd.nokia.landmarkcollection+xml460 1 application/vnd.nokia.ncd461 1 application/vnd.nokia.n-gage.ac+xml462 1 application/vnd.nokia.n-gage.data463 1 application/vnd.nokia.n-gage.symbian.install464 1 application/vnd.nokia.pcd+wbxml465 1 application/vnd.nokia.pcd+xml466 1 application/vnd.nokia.radio-preset467 1 application/vnd.nokia.radio-presets468 1 application/vnd.novadigm.EDM469 1 application/vnd.novadigm.EDX470 1 application/vnd.novadigm.EXT471 1 application/vnd.oasis.opendocument.chart472 1 application/vnd.oasis.opendocument.chart-template473 1 application/vnd.oasis.opendocument.database474 1 application/vnd.oasis.opendocument.formula475 1 application/vnd.oasis.opendocument.formula-template476 1 application/vnd.oasis.opendocument.graphics477 1 application/vnd.oasis.opendocument.graphics-template478 1 application/vnd.oasis.opendocument.image479 1 application/vnd.oasis.opendocument.image-template480 1 application/vnd.oasis.opendocument.presentation481 1 application/vnd.oasis.opendocument.presentation-template482 1 application/vnd.oasis.opendocument.spreadsheet483 1 application/vnd.oasis.opendocument.spreadsheet-template484 1 application/vnd.oasis.opendocument.text485 1 application/vnd.oasis.opendocument.text-master486 1 application/vnd.oasis.opendocument.text-template487 1 application/vnd.oasis.opendocument.text-web488 1 application/vnd.obn489 1 application/vnd.olpc-sugar490 1 application/vnd.oma.bcast.associated-procedure-parameter+xml491 1 application/vnd.oma.bcast.drm-trigger+xml492 1 application/vnd.oma.bcast.imd+xml493 1 application/vnd.oma.bcast.ltkm494 1 application/vnd.oma.bcast.notification+xml495 1 application/vnd.oma.bcast.provisioningtrigger496 1 application/vnd.oma.bcast.sgboot497 1 application/vnd.oma.bcast.sgdd+xml498 1 application/vnd.oma.bcast.sgdu499 1 application/vnd.oma.bcast.simple-symbol-container500 1 application/vnd.oma.bcast.smartcard-trigger+xml501 1 application/vnd.oma.bcast.sprov+xml502 1 application/vnd.oma.bcast.stkm503 1 application/vnd.oma.dcd504 1 application/vnd.oma.dcdc505 1 application/vnd.oma.dd2+xml506 1 application/vnd.oma.drm.risd+xml507 1 application/vnd.oma.group-usage-list+xml508 1 application/vnd.oma.poc.detailed-progress-report+xml509 1 application/vnd.oma.poc.final-report+xml510 1 application/vnd.oma.poc.groups+xml511 1 application/vnd.oma.poc.invocation-descriptor+xml512 1 application/vnd.oma.poc.optimized-progress-report+xml513 1 application/vnd.oma.xcap-directory+xml514 1 application/vnd.omads-email+xml515 1 application/vnd.omads-file+xml516 1 application/vnd.omads-folder+xml517 1 application/vnd.omaloc-supl-init518 1 application/vnd.oma-scws-config519 1 application/vnd.oma-scws-http-request520 1 application/vnd.oma-scws-http-response521 1 application/vnd.openofficeorg.extension522 1 application/vnd.osa.netdeploy523 1 application/vnd.osgi.bundle524 1 application/vnd.osgi.dp525 1 application/vnd.otps.ct-kip+xml526 1 application/vnd.palm527 1 application/vnd.paos.xml528 1 application/vnd.pg.format529 1 application/vnd.pg.osasli530 1 application/vnd.piaccess.application-licence531 1 application/vnd.picsel532 1 application/vnd.poc.group-advertisement+xml533 1 application/vnd.pocketlearn534 1 application/vnd.powerbuilder6535 1 application/vnd.powerbuilder6-s536 1 application/vnd.powerbuilder7537 1 application/vnd.powerbuilder75538 1 application/vnd.powerbuilder75-s539 1 application/vnd.powerbuilder7-s540 1 application/vnd.preminet541 1 application/vnd.previewsystems.box542 1 application/vnd.proteus.magazine543 1 application/vnd.publishare-delta-tree544 1 application/vnd.pvi.ptid1545 1 application/vnd.pwg-multiplexed546 1 application/vnd.pwg-xhtml-print+xml547 1 application/vnd.qualcomm.brew-app-res548 1 application/vnd.Quark.QuarkXPress549 1 application/vnd.rapid550 1 application/vnd.recordare.musicxml551 1 application/vnd.recordare.musicxml+xml552 1 application/vnd.RenLearn.rlprint553 1 application/vnd.route66.link66+xml554 1 application/vnd.ruckus.download555 1 application/vnd.s3sms556 1 application/vnd.sbm.cid557 1 application/vnd.sbm.mid2558 1 application/vnd.scribus559 1 application/vnd.sealed.3df560 1 application/vnd.sealed.csf561 1 application/vnd.sealed.doc562 1 application/vnd.sealed.eml563 1 application/vnd.sealed.mht564 1 application/vnd.sealed.net565 1 application/vnd.sealed.ppt566 1 application/vnd.sealed.tiff567 1 application/vnd.sealed.xls568 1 application/vnd.sealedmedia.softseal.html569 1 application/vnd.sealedmedia.softseal.pdf570 1 application/vnd.seemail571 1 application/vnd.sema572 1 application/vnd.semd573 1 application/vnd.semf574 1 application/vnd.shana.informed.formdata575 1 application/vnd.shana.informed.formtemplate576 1 application/vnd.shana.informed.interchange577 1 application/vnd.shana.informed.package578 1 application/vnd.SimTech-MindMapper579 1 application/vnd.smaf580 1 application/vnd.smart.teacher581 1 application/vnd.software602.filler.form+xml582 1 application/vnd.software602.filler.form-xml-zip583 1 application/vnd.solent.sdkm+xml584 1 application/vnd.spotfire.dxp585 1 application/vnd.spotfire.sfs586 1 application/vnd.sss-cod587 1 application/vnd.sss-dtf588 1 application/vnd.sss-ntf589 1 application/vnd.street-stream590 1 application/vnd.sun.wadl+xml591 1 application/vnd.sus-calendar592 1 application/vnd.svd593 1 application/vnd.swiftview-ics594 1 application/vnd.syncml.dm+wbxml595 1 application/vnd.syncml.dm+xml596 1 application/vnd.syncml.ds.notification597 1 application/vnd.syncml+xml598 1 application/vnd.tao.intent-module-archive599 1 application/vnd.tmobile-livetv600 1 application/vnd.trid.tpt601 1 application/vnd.triscape.mxs602 1 application/vnd.trueapp603 1 application/vnd.truedoc604 1 application/vnd.ufdl605 1 application/vnd.uiq.theme606 1 application/vnd.umajin607 1 application/vnd.unity608 1 application/vnd.uoml+xml609 1 application/vnd.uplanet.alert610 1 application/vnd.uplanet.alert-wbxml611 1 application/vnd.uplanet.bearer-choice612 1 application/vnd.uplanet.bearer-choice-wbxml613 1 application/vnd.uplanet.cacheop614 1 application/vnd.uplanet.cacheop-wbxml615 1 application/vnd.uplanet.channel616 1 application/vnd.uplanet.channel-wbxml617 1 application/vnd.uplanet.list618 1 application/vnd.uplanet.listcmd619 1 application/vnd.uplanet.listcmd-wbxml620 1 application/vnd.uplanet.list-wbxml621 1 application/vnd.uplanet.signal622 1 application/vnd.vcx623 1 application/vnd.vd-study624 1 application/vnd.vectorworks625 1 application/vnd.vidsoft.vidconference626 1 application/vnd.visio627 1 application/vnd.visionary628 1 application/vnd.vividence.scriptfile629 1 application/vnd.vsf630 1 application/vnd.wap.sic631 1 application/vnd.wap.slc632 1 application/vnd.wap.wbxml633 1 application/vnd.wap.wmlc634 1 application/vnd.wap.wmlscriptc635 1 application/vnd.webturbo636 1 application/vnd.wfa.wsc637 1 application/vnd.wmc638 1 application/vnd.wmf.bootstrap639 1 application/vnd.wordperfect640 1 application/vnd.wqd641 1 application/vnd.wrq-hp3000-labelled642 1 application/vnd.wt.stf643 1 application/vnd.wv.csp+xml644 1 application/vnd.wv.csp+wbxml645 1 application/vnd.wv.ssp+xml646 1 application/vnd.xara647 1 application/vnd.xfdl648 1 application/vnd.xfdl.webform649 1 application/vnd.xmi+xml650 1 application/vnd.xmpie.cpkg651 1 application/vnd.xmpie.dpkg652 1 application/vnd.xmpie.plan653 1 application/vnd.xmpie.ppkg654 1 application/vnd.xmpie.xlim655 1 application/vnd.yamaha.hv-dic656 1 application/vnd.yamaha.hv-script657 1 application/vnd.yamaha.hv-voice658 1 application/vnd.yamaha.openscoreformat659 1 application/vnd.yamaha.smaf-audio660 1 application/vnd.yamaha.smaf-phrase661 1 application/vnd.yellowriver-custom-menu662 1 application/vnd.zul663 1 application/vnd.zzazz.deck+xml664 1 application/voicexml+xml665 1 application/watcherinfo+xml666 1 application/whoispp-query667 1 application/whoispp-response668 1 application/wita669 1 application/wordperfect5.1670 1 application/wsdl+xml671 1 application/wspolicy+xml672 1 application/x400-bp673 1 application/xcap-att+xml674 1 application/xcap-caps+xml675 1 application/xcap-el+xml676 1 application/xcap-error+xml677 1 application/xcap-ns+xml678 1 application/xcon-conference-info-diff+xml679 1 application/xcon-conference-info+xml680 1 application/xenc+xml681 1 application/xhtml-voice+xml (Obsolete)682 1 application/xhtml+xml683 1 application/xml684 1 application/xml-dtd685 1 application/xml-external-parsed-entity686 1 application/xmpp+xml687 1 application/xop+xml688 1 application/xv+xml689 1 application/zip690 2 audio/32kadpcm691 2 audio/3gpp692 2 audio/3gpp2693 2 audio/ac3694 2 audio/AMR695 2 audio/AMR-WB696 2 audio/amr-wb+697 2 audio/asc698 2 audio/basic699 2 audio/BV16700 2 audio/BV32701 2 audio/clearmode702 2 audio/CN703 2 audio/DAT12704 2 audio/dls705 2 audio/dsr-es201108706 2 audio/dsr-es202050707 2 audio/dsr-es202211708 2 audio/dsr-es202212709 2 audio/eac3710 2 audio/DVI4711 2 audio/EVRC712 2 audio/EVRC0713 2 audio/EVRC1714 2 audio/EVRCB715 2 audio/EVRCB0716 2 audio/EVRCB1717 2 audio/EVRC-QCP718 2 audio/EVRCWB719 2 audio/EVRCWB0720 2 audio/EVRCWB1721 2 audio/example722 2 audio/G719723 2 audio/G722724 2 audio/G7221725 2 audio/G723726 2 audio/G726-16727 2 audio/G726-24728 2 audio/G726-32729 2 audio/G726-40730 2 audio/G728731 2 audio/G729732 2 audio/G7291733 2 audio/G729D734 2 audio/G729E735 2 audio/GSM736 2 audio/GSM-EFR737 2 audio/iLBC738 2 audio/L8739 2 audio/L16740 2 audio/L20741 2 audio/L24742 2 audio/LPC743 2 audio/mobile-xmf744 2 audio/MPA745 2 audio/mp4746 2 audio/MP4A-LATM747 2 audio/mpa-robust748 2 audio/mpeg749 2 audio/mpeg4-generic750 2 audio/ogg751 2 audio/parityfec752 2 audio/PCMA753 2 audio/PCMA-WB754 2 audio/PCMU755 2 audio/PCMU-WB756 2 audio/prs.sid757 2 audio/QCELP758 2 audio/RED759 2 audio/rtp-enc-aescm128760 2 audio/rtp-midi761 2 audio/rtx762 2 audio/SMV763 2 audio/SMV0764 2 audio/SMV-QCP765 2 audio/sp-midi766 2 audio/t140c767 2 audio/t38768 2 audio/telephone-event769 2 audio/tone770 2 audio/ulpfec771 2 audio/VDVI772 2 audio/VMR-WB773 2 audio/vnd.3gpp.iufp774 2 audio/vnd.4SB775 2 audio/vnd.audiokoz776 2 audio/vnd.CELP777 2 audio/vnd.cisco.nse778 2 audio/vnd.cmles.radio-events779 2 audio/vnd.cns.anp1780 2 audio/vnd.cns.inf1781 2 audio/vnd.digital-winds782 2 audio/vnd.dlna.adts783 2 audio/vnd.dolby.heaac.1784 2 audio/vnd.dolby.mlp785 2 audio/vnd.dolby.mps786 2 audio/vnd.dts787 2 audio/vnd.dts.hd788 2 audio/vnd.everad.plj789 2 audio/vnd.hns.audio790 2 audio/vnd.lucent.voice791 2 audio/vnd.ms-playready.media.pya792 2 audio/vnd.nokia.mobile-xmf793 2 audio/vnd.nortel.vbk794 2 audio/vnd.nuera.ecelp4800795 2 audio/vnd.nuera.ecelp7470796 2 audio/vnd.nuera.ecelp9600797 2 audio/vnd.octel.sbc798 2 audio/vnd.qcelp - DEPRECATED - Please use audio/qcelp799 2 audio/vnd.rhetorex.32kadpcm800 2 audio/vnd.sealedmedia.softseal.mpeg801 2 audio/vnd.vmx.cvsd802 2 audio/vorbis803 2 audio/vorbis-config804 4 image/cgm805 4 image/example806 4 image/fits807 4 image/g3fax808 4 image/gif809 4 image/ief810 4 image/jp2811 4 image/jpeg812 4 image/jpm813 4 image/jpx814 4 image/naplps815 4 image/png816 4 image/prs.btif817 4 image/prs.pti818 4 image/t38819 4 image/tiff820 4 image/tiff-fx821 4 image/vnd.adobe.photoshop822 4 image/vnd.cns.inf2823 4 image/vnd.djvu824 4 image/vnd.dwg825 4 image/vnd.dxf826 4 image/vnd.fastbidsheet827 4 image/vnd.fpx828 4 image/vnd.fst829 4 image/vnd.fujixerox.edmics-mmr830 4 image/vnd.fujixerox.edmics-rlc831 4 image/vnd.globalgraphics.pgb832 4 image/vnd.microsoft.icon833 4 image/vnd.mix834 4 image/vnd.ms-modi835 4 image/vnd.net-fpx836 4 image/vnd.radiance837 4 image/vnd.sealed.png838 4 image/vnd.sealedmedia.softseal.gif839 4 image/vnd.sealedmedia.softseal.jpg840 4 image/vnd.svf841 4 image/vnd.wap.wbmp842 4 image/vnd.xiff843 5 message/CPIM844 5 message/delivery-status845 5 message/disposition-notification846 5 message/example847 5 message/external-body848 5 message/global849 5 message/global-delivery-status850 5 message/global-disposition-notification851 5 message/global-headers852 5 message/http853 5 message/imdn+xml854 5 message/news855 5 message/partial856 5 message/rfc822857 5 message/s-http858 5 message/sip859 5 message/sipfrag860 5 message/tracking-status861 5 message/vnd.si.simp862 6 model/example863 6 model/iges864 6 model/mesh865 6 model/vnd.dwf866 6 model/vnd.flatland.3dml867 6 model/vnd.gdl868 6 model/vnd.gs-gdl869 6 model/vnd.gtw870 6 model/vnd.moml+xml871 6 model/vnd.mts872 6 model/vnd.parasolid.transmit.binary873 6 model/vnd.parasolid.transmit.text874 6 model/vnd.vtu875 6 model/vrml876 7 multipart/alternative877 7 multipart/appledouble878 7 multipart/byteranges879 7 multipart/digest880 7 multipart/encrypted881 7 multipart/example882 7 multipart/form-data883 7 multipart/header-set884 7 multipart/mixed885 7 multipart/parallel886 7 multipart/related887 7 multipart/report888 7 multipart/signed889 7 multipart/voice-message890 8 text/calendar891 8 text/css892 8 text/csv893 8 text/directory894 8 text/dns895 8 text/ecmascript (obsolete)896 8 text/enriched897 8 text/example898 8 text/html899 8 text/javascript (obsolete)900 8 text/parityfec901 8 text/plain902 8 text/prs.fallenstein.rst903 8 text/prs.lines.tag904 8 text/RED905 8 text/rfc822-headers906 8 text/richtext907 8 text/rtf908 8 text/rtp-enc-aescm128909 8 text/rtx910 8 text/sgml911 8 text/t140912 8 text/tab-separated-values913 8 text/troff914 8 text/ulpfec915 8 text/uri-list916 8 text/vnd.abc917 8 text/vnd.curl918 8 text/vnd.DMClientScript919 8 text/vnd.esmertec.theme-descriptor920 8 text/vnd.fly921 8 text/vnd.fmi.flexstor922 8 text/vnd.graphviz923 8 text/vnd.in3d.3dml924 8 text/vnd.in3d.spot925 8 text/vnd.IPTC.NewsML926 8 text/vnd.IPTC.NITF927 8 text/vnd.latex-z928 8 text/vnd.motorola.reflex929 8 text/vnd.ms-mediapackage930 8 text/vnd.net2phone.commcenter.command931 8 text/vnd.si.uricatalogue932 8 text/vnd.sun.j2me.app-descriptor933 8 text/vnd.trolltech.linguist934 8 text/vnd.wap.si935 8 text/vnd.wap.sl936 8 text/vnd.wap.wml937 8 text/vnd.wap.wmlscript938 8 text/xml939 8 text/xml-external-parsed-entity940 9 video/3gpp941 9 video/3gpp2942 9 video/3gpp-tt943 9 video/BMPEG944 9 video/BT656945 9 video/CelB946 9 video/DV947 9 video/example948 9 video/H261949 9 video/H263950 9 video/H263-1998951 9 video/H263-2000952 9 video/H264953 9 video/JPEG954 9 video/jpeg2000955 9 video/MJ2956 9 video/MP1S957 9 video/MP2P958 9 video/MP2T959 9 video/mp4960 9 video/MP4V-ES961 9 video/MPV962 9 video/mpeg963 9 video/mpeg4-generic964 9 video/nv965 9 video/ogg966 9 video/parityfec967 9 video/pointer968 9 video/quicktime969 9 video/raw970 9 video/rtp-enc-aescm128971 9 video/rtx972 9 video/SMPTE292M973 9 video/ulpfec974 9 video/vc1975 9 video/vnd.CCTV976 9 video/vnd.dlna.mpeg-tts977 9 video/vnd.fvt978 9 video/vnd.hns.video979 9 video/vnd.iptvforum.1dparityfec-1010980 9 video/vnd.iptvforum.1dparityfec-2005981 9 video/vnd.iptvforum.2dparityfec-1010982 9 video/vnd.iptvforum.2dparityfec-2005983 9 video/vnd.iptvforum.ttsavc984 9 video/vnd.iptvforum.ttsmpeg2985 9 video/vnd.motorola.video986 9 video/vnd.motorola.videop987 9 video/vnd.mpegurl988 9 video/vnd.ms-playready.media.pyv989 9 video/vnd.nokia.interleaved-multimedia990 9 video/vnd.nokia.videovoip991 9 video/vnd.objectvideo992 9 video/vnd.sealed.mpeg1993 9 video/vnd.sealed.mpeg4994 9 video/vnd.sealed.swf995 9 video/vnd.sealedmedia.softseal.mov996 9 video/vnd.vivo997 6 image/x-icon998 4 image/pjpeg999 4 image/x-xbitmap1000 1 application/x-shockwave-flash1001 4 image/svg+xml1002 4 image/jpg
Mike
For best service when you require assistance:
Skype: arachnodedotnet
Another condition that can occur that throw a 406 is when you ask for file such as a GIF:
But we actually get a JAVASCRIPT back, which be default, isn't allowed.
The reason for the bait and switch could be tracking, or to save bandwidth for relatively unfamiliar crawlers, or due to a coding error.
With a bit of code to check for another AbsoluteUri responding that differs from the WebClient.RequestUri, we don't see the 406 problem.
For a reason that I currently don't know, .gifs are more prone to redirection, and in the default configuration of arachnode.net, to an unsupported AllowedDataType.
-Mike
I have commented a line in DataManager.cs (to be checked in) that sets the Accept property of the WebClient. I need to do some additional digging to find out why .gifs and .css files are often rejected.
However, since AN will now make HEAD requests before downloading the content stream, we can check for AllowedDataTypes. More to come...
this is exactly who this guy solved it out:
developer.yahoo.net/forum/index.php?showtopic=76
Thanks.