arachnode.net
An Open Source C# web crawler with Lucene.NET search using SQL Server 2008/2012/2014/2016/CE An Open Source C# web crawler with Lucene.NET search using MongoDB/RavenDB/Hadoop
Search the Live Index Does arachnode.net scale? | Download the latest release

The request was aborted: The connection was closed unexpectedly.

rated by 0 users
Answered (Verified) This post has 2 verified answers | 7 Replies | 2 Followers

Top 10 Contributor
229 Posts
megetron posted on Tue, Aug 24 2010 3:06 PM

Hello,

I am getting this:

The request was aborted: The connection was closed unexpectedly. System    at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)     at Arachnode.SiteCrawler.Components.WebClient.DownloadData(String absoluteUri) in E:\DEVELOPMENT\2.5\SiteCrawler\Components\WebClient.cs:line 238

 

What can it be?

Answered (Verified) Verified Answer

Top 10 Contributor
1,905 Posts
Verified by megetron

This comes from the Web servers themselves.

  • Starred results for The connection was closed unexpectedly

    1. Forums - arachnode.net - arachnode.net/forums/
  • Exception: "The underlying connection was closed: The connection ...

    24 posts - 10 authors - Last post: Jan 18, 2006 As server closes the connection .Net displays The connection was closed unexpectedly. But according to the design of server its expected for ...
    social.msdn.microsoft.com/.../246ffc07-1cab-44b5-b529-f1135866ebca/ - Cached The underlying connection was closed: The connection was closed ...
    "The connection was closed unexpectedly" errors when trying to ...

    More results from social.msdn.microsoft.com »

  • The underlying connection was closed: The connection was closed ...

    Jun 16, 2010 ... This exception is consistently thrown on a SOAP Request which takes ... I believe this problem is related to load balanced servers. – ...
    stackoverflow.com/.../the-underlying-connection-was-closed-the-connection-was-closed-unexpectedly - Cached - Similar
  • Exchange/SMTP connection was unexpectedly closed or error message ...

    Exchange/SMTP connection was unexpectedly closed or error message "454.5.7.3 Client does not have permission to submit mail to this server." ...
    www.servolutions.com/support/articles/connectionclosed.htm - Cached
  • RealVNC - How to find the VNC® Server error log

    Dec 22, 2005 ... Hence, the error message Connection closed unexpectedly does not provide enough information to diagnose your problem. ...
    www.realvnc.com/support/serverlog.html - Cached - Similar
  • RealVNC - Frequently asked questions

    What does 'Connection closed unexpectedly' mean? Why can I access my VNC Server even though I'm entering the wrong password? Why can't I access my VNC ...
    www.realvnc.com/support/faq.html - Cached - Similar Show more results from www.realvnc.com

     

  • Frequent Cause of WCF Exception "The connection was closed ...

    Jul 29, 2009 ... Sidar Ok wrote re: Frequent Cause of WCF Exception "The connection was closed unexpectedly". on 08-07-2009 5:46 AM. Nice one Billy. ...
    devlicio.us/.../frequent-cause-of-wcf-exception-quot-the-connection-was-closed-unexpectedly-quot.aspx - Cached - Similar
  • WCF: The connection was closed unexpectedly « Mint

    Jan 20, 2010 ... CommunicationException: The underlying connection was closed: The connection was closed unexpectedly. In your frustration of pulling your ...
    mint.litemedia.se/.../wcf-the-connection-was-closed-unexpectedly/ - Cached - Similar
  • The connection was closed unexpectedly (unknown listener event: 0 ...

    Jun 5, 2005 ... The connection was closed unexpectedly (unknown listener event: 0) ... error "The connection closed unexpectedly" and the event log contains ...
    www.gossamer-threads.com/lists/vnc/list/51153 - Cached - Similar The Connection Closed Unexpectedly‎ - Feb 23, 2009
    Connection closed unexpectedly.‎ - Oct 15, 2005

    More results from gossamer-threads.com »

  • WCF Client Error “The connection was closed unexpectedly” Calling ...

    Jan 23, 2010 ... WCF Client Error “The connection was closed unexpectedly” Calling Java/WebSphere 7 Web Service.
    www.chinhdo.com/20100123/wcf-java-100-continue/ - Cached - Similar
  • RE: "The connection closed unexpectedly"

    RE: "The connection closed unexpectedly". James Weatherall Thu, 26 Jan 2006 03:02:04 -0800. Ben, Are you using Fast User Switching or Remote Desktop on the ...
    [email protected]../msg21593.html - Cached - Similar
  • For best service when you require assistance:

    1. Check the DisallowedAbsoluteUris and Exceptions tables first.
    2. Cut and paste actual exceptions from the Exceptions table.
    3. Include screenshots.

    Skype: arachnodedotnet

    Top 10 Contributor
    1,905 Posts
    Verified by megetron

    Also, in this case, it's not so much an error, but you trying to outwit a site that doesn't want you to crawl them.  Stick out tongue

    Check Program.cs.  This is likely overriding your threads config setting.

    For best service when you require assistance:

    1. Check the DisallowedAbsoluteUris and Exceptions tables first.
    2. Cut and paste actual exceptions from the Exceptions table.
    3. Include screenshots.

    Skype: arachnodedotnet

    All Replies

    Top 10 Contributor
    1,905 Posts
    Verified by megetron

    This comes from the Web servers themselves.

  • Starred results for The connection was closed unexpectedly

    1. Forums - arachnode.net - arachnode.net/forums/
  • Exception: "The underlying connection was closed: The connection ...

    24 posts - 10 authors - Last post: Jan 18, 2006 As server closes the connection .Net displays The connection was closed unexpectedly. But according to the design of server its expected for ...
    social.msdn.microsoft.com/.../246ffc07-1cab-44b5-b529-f1135866ebca/ - Cached The underlying connection was closed: The connection was closed ...
    "The connection was closed unexpectedly" errors when trying to ...

    More results from social.msdn.microsoft.com »

  • The underlying connection was closed: The connection was closed ...

    Jun 16, 2010 ... This exception is consistently thrown on a SOAP Request which takes ... I believe this problem is related to load balanced servers. – ...
    stackoverflow.com/.../the-underlying-connection-was-closed-the-connection-was-closed-unexpectedly - Cached - Similar
  • Exchange/SMTP connection was unexpectedly closed or error message ...

    Exchange/SMTP connection was unexpectedly closed or error message "454.5.7.3 Client does not have permission to submit mail to this server." ...
    www.servolutions.com/support/articles/connectionclosed.htm - Cached
  • RealVNC - How to find the VNC® Server error log

    Dec 22, 2005 ... Hence, the error message Connection closed unexpectedly does not provide enough information to diagnose your problem. ...
    www.realvnc.com/support/serverlog.html - Cached - Similar
  • RealVNC - Frequently asked questions

    What does 'Connection closed unexpectedly' mean? Why can I access my VNC Server even though I'm entering the wrong password? Why can't I access my VNC ...
    www.realvnc.com/support/faq.html - Cached - Similar Show more results from www.realvnc.com

     

  • Frequent Cause of WCF Exception "The connection was closed ...

    Jul 29, 2009 ... Sidar Ok wrote re: Frequent Cause of WCF Exception "The connection was closed unexpectedly". on 08-07-2009 5:46 AM. Nice one Billy. ...
    devlicio.us/.../frequent-cause-of-wcf-exception-quot-the-connection-was-closed-unexpectedly-quot.aspx - Cached - Similar
  • WCF: The connection was closed unexpectedly « Mint

    Jan 20, 2010 ... CommunicationException: The underlying connection was closed: The connection was closed unexpectedly. In your frustration of pulling your ...
    mint.litemedia.se/.../wcf-the-connection-was-closed-unexpectedly/ - Cached - Similar
  • The connection was closed unexpectedly (unknown listener event: 0 ...

    Jun 5, 2005 ... The connection was closed unexpectedly (unknown listener event: 0) ... error "The connection closed unexpectedly" and the event log contains ...
    www.gossamer-threads.com/lists/vnc/list/51153 - Cached - Similar The Connection Closed Unexpectedly‎ - Feb 23, 2009
    Connection closed unexpectedly.‎ - Oct 15, 2005

    More results from gossamer-threads.com »

  • WCF Client Error “The connection was closed unexpectedly” Calling ...

    Jan 23, 2010 ... WCF Client Error “The connection was closed unexpectedly” Calling Java/WebSphere 7 Web Service.
    www.chinhdo.com/20100123/wcf-java-100-continue/ - Cached - Similar
  • RE: "The connection closed unexpectedly"

    RE: "The connection closed unexpectedly". James Weatherall Thu, 26 Jan 2006 03:02:04 -0800. Ben, Are you using Fast User Switching or Remote Desktop on the ...
    [email protected]../msg21593.html - Cached - Similar
  • For best service when you require assistance:

    1. Check the DisallowedAbsoluteUris and Exceptions tables first.
    2. Cut and paste actual exceptions from the Exceptions table.
    3. Include screenshots.

    Skype: arachnodedotnet

    Top 10 Contributor
    229 Posts

    Yes you right. my mistake. should follow server instructions.

    thanks

     

    Top 10 Contributor
    229 Posts

    How can I know what is the reason for that? I am using IE7 and I can see  a page, but when trying to crawl the same page, I get the message...

    I even replace IP, but stil the same.

    Please advice, thank you,

    Top 10 Contributor
    1,905 Posts

    Try setting the UserAgent to a browser UserAgent.

    Also, many web servers will only allow so many concurrent connections from any one IP.

    This is a good post on server behavior: http://arachnode.net/blogs/arachnode_net/archive/2010/04/29/troubleshooting-crawl-result-differences-between-different-crawl-environments.aspx

    For best service when you require assistance:

    1. Check the DisallowedAbsoluteUris and Exceptions tables first.
    2. Cut and paste actual exceptions from the Exceptions table.
    3. Include screenshots.

    Skype: arachnodedotnet

    Top 10 Contributor
    1,905 Posts

    Nice link set:

    List of User Agent Strings

    ALL 

    CRAWLERS

    ABACHOBot
    Accoona-AI-Agent
    AnyApexBot
    Arachmo
    B-l-i-t-z-B-O-T
    Baiduspider
    BecomeBot
    Bimbot
    BlitzBOT
    boitho.com-dc
    boitho.com-robot
    btbot
    Cerberian Drtrs
    Charlotte
    ConveraCrawler
    cosmos
    DataparkSearch
    DiamondBot
    Discobot
    Dotbot
    EmeraldShield.com WebBot
    envolk[ITS]spider
    EsperanzaBot
    Exabot
    FAST Enterprise Crawler
    FAST-WebCrawler
    FDSE robot
    FindLinks
    FurlBot
    FyberSpider
    g2crawler
    Gaisbot
    GalaxyBot
    genieBot
    Gigabot
    Girafabot
    Googlebot
    Googlebot-Image
    hl_ftien_spider
    htdig
    ia_archiver
    ichiro
    IRLbot
    IssueCrawler
    Jyxobot
    LapozzBot
    Larbin
    LinkWalker
    lmspider
    lwp-trivial
    mabontland
    magpie-crawler
    Mediapartners-Google
    MJ12bot
    Mnogosearch
    mogimogi
    MojeekBot
    Morning Paper
    msnbot
    MSRBot
    MVAClient
    NetResearchServer
    NG-Search
    nicebot
    noxtrumbot
    Nusearch Spider
    NutchCVS
    obot
    oegp
    OmniExplorer_Bot
    OOZBOT
    Orbiter
    PageBitesHyperBot
    polybot
    Pompos
    Psbot
    PycURL
    RAMPyBot
    RufusBot
    SandCrawler
    SBIder
    Scrubby
    SearchSight
    Seekbot
    semanticdiscovery
    Sensis Web Crawler
    SEOChat::Bot
    Shim-Crawler
    ShopWiki
    Shoula robot
    silk
    Snappy
    sogou spider
    Speedy Spider
    Sqworm
    StackRambler
    SurveyBot
    SynooBot
    Teoma
    TerrawizBot
    TheSuBot
    Thumbnail.CZ robot
    TinEye
    TurnitinBot
    updated
    Vagabondo
    VoilaBot
    Vortex
    voyager
    VYU2
    webcollage
    Websquash.com
    wf84
    WoFindeIch Robot
    Xaldon_WebSpider
    yacy
    Yahoo! Slurp
    Yahoo! Slurp China
    YahooSeeker
    YahooSeeker-Testing
    YandexBot
    yoogliFetchAgent
    Zao
    Zealbot
    zspider
    ZyBorg
    BROWSERS

    ABrowse
    Acoo Browser
    America Online Browser
    AmigaVoyager
    AOL
    Arora
    Avant Browser
    Beonex
    BonEcho
    Camino
    Charon
    Cheshire
    Chimera
    Chrome
    ChromePlus
    CometBird
    Crazy Browser
    Cyberdog
    Deepnet Explorer
    DeskBrowse
    Dillo
    Element Browser
    Elinks
    Enigma Browser
    Epiphany
    Escape
    Fennec
    Firebird
    Firefox
    Flock
    Fluid
    Galaxy
    Galeon
    GranParadiso
    GreenBrowser
    Hana
    HotJava
    IBM WebExplorer
    IBrowse
    iCab
    Iceape
    IceCat
    Iceweasel
    iNet Browser
    Internet Explorer
    iRider
    Iron
    K-Meleon
    K-Ninja
    Kapiko
    Kazehakase
    KKman
    KMLite
    Konqueror
    LeechCraft
    Links
    Lobo
    lolifox
    Lorentz
    Lunascape
    Lynx
    Madfox
    Maxthon
    Midori
    Minefield
    Minimo
    Mozilla
    MultiZilla
    myibrow
    MyIE2
    Namoroka
    NCSA_Mosaic
    NetFront
    NetNewsWire
    NetPositive
    Netscape
    NetSurf
    OmniWeb
    Opera
    Opera Mini
    Opera Mobi
    Orca
    Oregano
    Palemoon
    Phoenix
    Pogo
    Prism
    QtWeb Internet Browser
    retawq
    Safari
    SeaMonkey
    Shiira
    Shiretoko
    Sleipnir
    SlimBrowser
    Stainless
    Sunrise
    TeaShark
    uZard Web
    uzbl
    Vonkeror
    w3m
    WorldWideWeb
    Wyzo

    CONSOLES

    libnup
    Playstation 3
    Playstation Portable
    Wii

    OFFLINE BROWSERS

    Offline Explorer
    SuperBot
    Web Downloader
    WebCopier
    WebZIP
    Wget

    E-MAIL CLIENTS

    Thunderbird
    LINK CHECKERS

    AbiLogicBot
    Link Valet
    Link Validity Check
    LinksManager.com_bot
    Mojoo Robot
    Notifixious
    online link validator
    Ploetz + Zeller
    Reciprocal Link System PRO
    REL Link Checker Lite
    SiteBar
    Vivante Link Checker
    W3C-checklink
    Xenu Link Sleuth

    E-MAIL COLLECTORS

    EmailSiphon

    VALIDATORS

    CSE HTML Validator
    CSSCheck
    Cynthia
    HTMLParser
    P3P Validator
    W3C_CSS_Validator_JFouffa
    W3C_Validator
    WDG_Validator

    FEED READERS

    Bloglines
    everyfeed-spider
    FeedFetcher-Google
    Gregarius

    LIBRARIES

    Java
    libwww-perl
    Peach
    Python-urllib

    OTHERS

    !Susie
    Amaya
    Cocoal.icio.us
    DomainsDB.net MetaCrawler
    GSiteCrawler
    Snoopy
    URD-MAGPIE
    Windows-Media-Player

    For best service when you require assistance:

    1. Check the DisallowedAbsoluteUris and Exceptions tables first.
    2. Cut and paste actual exceptions from the Exceptions table.
    3. Include screenshots.

    Skype: arachnodedotnet

    Top 10 Contributor
    229 Posts

    Hi, after investigating this further:

    1, rename user agent several useragents of IE.
    2. Changing the webclient.Cs headrs:

                 HttpWebRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
                HttpWebRequest.Headers.Add("UA-CPU", "x86");
                HttpWebRequest.Headers.Add("Accept-Language", "en-us");
                HttpWebRequest.Headers.Add("Pragma", "no-cache");

     

    but still the error exists...

    So I used fiddler and I find that the page is downloaded partially and stops in the middle. the webserver closes connection.

    Now I tried to crawl only one thread in frequence of 10 seconds, and still happens.

    Funny thing is that when debug the appllication it's seems like there are 10 threads running and 1 as I defined in the configuration table...

    :\

     

     

    Top 10 Contributor
    1,905 Posts
    Verified by megetron

    Also, in this case, it's not so much an error, but you trying to outwit a site that doesn't want you to crawl them.  Stick out tongue

    Check Program.cs.  This is likely overriding your threads config setting.

    For best service when you require assistance:

    1. Check the DisallowedAbsoluteUris and Exceptions tables first.
    2. Cut and paste actual exceptions from the Exceptions table.
    3. Include screenshots.

    Skype: arachnodedotnet

    Page 1 of 1 (8 items) | RSS
    An Open Source C# web crawler with Lucene.NET search using SQL 2008/2012/CE

    copyright 2004-2017, arachnode.net LLC