Asked By someone
22-Apr-08 04:45 PM
Let me start by giving you the history of my application:
It used to run perfectly fine on my old Windows 2000 box on IIS5 with MSSQL 7.
It was time for an upgrade so we purchased a new Win2k3 x64 box running IIS6
and MSSQL 2005 Express. We transferred databases successfully, the website,
and COM objects (one had to be recoded to be x64 compatible).
Now, before going live with the new site, we tested EVERY page of the site,
so we know all database communication works, all COM objects do what they
need to do, and nothing breaks anywhere (if it does i don't see an errors
displayed on the page).
We went live with the new server, and eventually noticed that IIS hangs.
When we go onto the server, there are no errors on the screen, IIS seems to
be running fine, and nothing in the EVENT Viewer (besides some time sync
errors). We tried to do some of our own troubleshooting and debugging, but i
guess we don't know what to look for or the proper way to debug. Everything
runs fine isolated, but when put live, the server will "hang" when loading
pages (as in the page will never load). No Server 500 error or any ASP
errors, just won't load.
After reading some pages here, i though it might be my DSN connection
string, i tried OBDC drivers and OLE DB, both didn't fix the issue.
Well, this can't keep happening but we have NO idea on what else to try! To
get the site working again, we can't run IISRESET (last time i did that i had
to revert everything to x32 mode to get it working again). So in x64 or x32
mode, IIS still freezes. To get it working, i just simple log off and log
back in. If i stay logged out, the ASP pages won't load (something about
creating an object failing, i'm guessing that the COM objects are set up to
use the current logged in user authentications to run). This leads me to
believe that the issue is related to our COM objects, but we have no sure way
of knowing WHICH one or how to even diagnose that.
The ASP pages are very old, coded in VB. I just installed IISState and ran a
dump on the last hang up.
What's weird about this log is that when i ran it, the site wasn't working,
and the dump seemed to almost pause for a bit, then it suddenly started
going. After the dump, i noticed the site was working again, but this NEVER
happens unless i log off first, so i'm not sure if this log even has all the
information needed since it might be a dump of the site working, not hanged
up.
Here it is:
Opened log file 'C:\iisstate\output\IISState-908.log'
***********************
Starting new log output
IISState version 3.3.1
Tue Apr 22 16:00:44 2008
OS = Windows 2003 Server
Executable: w3wp.exe
PID = 908
Note: Thread times are formatted as HH:MM:SS.ms
***********************
Thread ID: 0
System Thread ID: 2dc
Kernel Time: 0:0:0.31
User Time: 0:0:0.62
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 0014fc0c 7d4d8c82 ntdll+0x1c828
01 0014fc7c 7d4d8bf1 kernel32!WaitForSingleObjectEx+0xac
02 0014fc90 5a36467a kernel32!WaitForSingleObject+0x12
03 0014fca0 5a366e63 w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
04 0014fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
05 0014fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
06 0014ff0c 0100187c w3core!UlW3Start+0x26e
07 0014ff44 01001a27 w3wp!wmain+0x22a
08 0014ffc0 7d4e7d2a w3wp!wmainCRTStartup+0x12f
09 0014fff0 00000000 kernel32!BaseProcessStart+0x28
Thread ID: 1
System Thread ID: e9c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0114ffa0 7d634d69 ntdll+0x1cca8
01 0114ffb8 7d4dfe21 ntdll+0x34d69
02 0114ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 2
System Thread ID: 4bc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 011cff74 7d62e139 ntdll+0x1c8a0
01 011cffb8 7d4dfe21 ntdll+0x2e139
02 011cffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 3
System Thread ID: c4c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: Unable to locate ASP page
No remote call being made
# ChildEBP RetAddr
00 0126fd1c 7da3da80 ntdll+0x1cbcd
01 0126ff84 7da45eac RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
02 0126ff8c 7da45dd0 RPCRT4!RecvLotsaCallsWrapper+0xd
03 0126ffac 7da45e94 RPCRT4!BaseCachedThreadRoutine+0x9d
04 0126ffb8 7d4dfe21 RPCRT4!ThreadStartRoutine+0x1b
05 0126ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 4
System Thread ID: bb8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 014eff28 7d5066b4 ntdll+0x1c8a0
01 014eff54 5a30249e kernel32!GetQueuedCompletionStatus+0x29
02 014eff8c 5a3026bc W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
03 014effa0 5a301db9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
04 014effb8 7d4dfe21 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
05 014effec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 5
System Thread ID: 30c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0156ff28 7d5066b4 ntdll+0x1c8a0
01 0156ff54 5a30249e kernel32!GetQueuedCompletionStatus+0x29
02 0156ff8c 5a3026bc W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
03 0156ffa0 5a301db9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
04 0156ffb8 7d4dfe21 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
05 0156ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 6
System Thread ID: 978
Kernel Time: 0:0:0.15
User Time: 0:0:0.46
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 015eff28 7d5066b4 ntdll+0x1c8a0
01 015eff54 5a30249e kernel32!GetQueuedCompletionStatus+0x29
02 015eff8c 5a3026bc W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
03 015effa0 5a301db9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
04 015effb8 7d4dfe21 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
05 015effec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 7
System Thread ID: a38
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0166ff28 7d5066b4 ntdll+0x1c8a0
01 0166ff54 5a30249e kernel32!GetQueuedCompletionStatus+0x29
02 0166ff8c 5a3026bc W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
03 0166ffa0 5a301db9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
04 0166ffb8 7d4dfe21 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
05 0166ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 8
System Thread ID: d34
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0176fcf0 7d642028 ntdll+0x1d051
01 0176ffb8 7d4dfe21 ntdll+0x42028
02 0176ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 9
System Thread ID: eb4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
\\?\C:\IISDebugTools\IISCHAgent.dll -
Thread Type: Other
# ChildEBP RetAddr
00 01befedc 7d5066b4 ntdll+0x1c8a0
01 01beff08 67306fab kernel32!GetQueuedCompletionStatus+0x29
WARNING: Stack unwind information not available. Following frames may be
wrong.
02 01beffb8 7d4dfe21 IISCHAgent!ConfigFileMonitor+0x15b
03 01beffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 10
System Thread ID: 618
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 01d6fec0 7d4e286c ntdll+0x1d051
01 01d6ff68 7d4e3e8e kernel32!WaitForMultipleObjectsEx+0x11a
02 01d6ff84 6730649c kernel32!WaitForMultipleObjects+0x18
WARNING: Stack unwind information not available. Following frames may be
wrong.
03 01d6ffb8 7d4dfe21 IISCHAgent!MonitorWorkerProcess+0xa3
04 01d6ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 11
System Thread ID: d4c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 0277fa88 7d4d8c82 ntdll+0x1c828
01 0277faf8 7d4d8bf1 kernel32!WaitForSingleObjectEx+0xac
02 0277fb0c 5a3b820f kernel32!WaitForSingleObject+0x12
03 0277ffb8 7d4dfe21 w3core!HTTP_COMPRESSION::CompressionThread+0x126
04 0277ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 12
System Thread ID: ad4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0287ff90 7d64240d ntdll+0x1cca8
01 0287ffb8 7d4dfe21 ntdll+0x4240d
02 0287ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 13
System Thread ID: 6a0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: ASP
Executing Page: Unable to locate ASP page
# ChildEBP RetAddr
00 0347ff10 7d4d8c82 ntdll+0x1c828
01 0347ff80 7d4d8bf1 kernel32!WaitForSingleObjectEx+0xac
02 0347ff94 709fc843 kernel32!WaitForSingleObject+0x12
03 0347ffb4 709fc81b asp!CApplnCleanupMgr::ApplnCleanupDoWork+0x1e
04 0347ffb8 7d4dfe21 asp!CApplnCleanupMgr::ApplnCleanupThread+0xa
05 0347ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 14
System Thread ID: e4c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: ASP
Executing Page: Unable to locate ASP page
# ChildEBP RetAddr
00 035efe20 7d4e286c ntdll+0x1d051
01 035efec8 7d94d299 kernel32!WaitForMultipleObjectsEx+0x11a
02 035eff24 7d94d327 USER32!RealMsgWaitForMultipleObjectsEx+0x152
03 035eff40 709fb8fe USER32!MsgWaitForMultipleObjects+0x1f
04 035eff84 77bcb530 asp!CMTACallbackThread::Thread+0x4f
05 035effb8 7d4dfe21 msvcrt!_endthreadex+0xa3
06 035effec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 15
System Thread ID: 344
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: ASP
Executing Page: Unable to locate ASP page
# ChildEBP RetAddr
00 0369fe20 7d4e286c ntdll+0x1d051
01 0369fec8 7d94d299 kernel32!WaitForMultipleObjectsEx+0x11a
02 0369ff24 7d94d327 USER32!RealMsgWaitForMultipleObjectsEx+0x152
03 0369ff40 70a01811 USER32!MsgWaitForMultipleObjects+0x1f
04 0369ff84 77bcb3ca asp!Debugger+0x9e
05 0369ffb8 7d4dfe21 msvcrt!_endthread+0xab
06 0369ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 16
System Thread ID: a0c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: PDM (Debugger) Thread.
# ChildEBP RetAddr
00 0372fd80 7d4e286c ntdll+0x1d051
01 0372fe28 7d94d299 kernel32!WaitForMultipleObjectsEx+0x11a
02 0372fe84 7d94d327 USER32!RealMsgWaitForMultipleObjectsEx+0x152
03 0372fea0 3f1a29b7 USER32!MsgWaitForMultipleObjects+0x1f
04 0372ff84 3f1b754b pdm+0x29b7
05 0372ffac 3f1b74de pdm+0x1754b
06 0372ffb8 7d4dfe21 pdm+0x174de
07 0372ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 17
System Thread ID: b64
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: ASP
Executing Page: Unable to locate ASP page
# ChildEBP RetAddr
00 0384fef8 7d4d8c82 ntdll+0x1c828
01 0384ff68 7d4d8bf1 kernel32!WaitForSingleObjectEx+0xac
02 0384ff7c 709fe91f kernel32!WaitForSingleObject+0x12
03 0384ffb8 7d4dfe21 asp!CViperReqManager::WatchThread+0x68
04 0384ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 18
System Thread ID: ec0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 19
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 20
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.78
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 21
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 22
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 23
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 24
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 25
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 26
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 27
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.15
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 28
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 29
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 30
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 31
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 32
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 33
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 34
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 35
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 36
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
Thread ID: 37
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 038eff5c 00000000 ntdll+0x1c828
*****
Dump name is formatted as: PID-Timestamp.dmp
Creating C:\iisstate\output\908-1208894577.dmp - mini user dump
GenGetProcessInfo.Start(0x38c) failed, 0x80070005
Dump creation failed, Win32 error 5
*****
Closing open log file C:\iisstate\output\IISState-908.log
I'd appreciate ANY help that can be given! If you don't see anything wrong
with this log, i'll try to dump another one when the site freezes again (i
just ran "iisstate.exe -p (pid) -d" i hope that's right)
Thanks in advanced
-Prateek