I'm the server administrator for a small business and I have one app pool
that is very problematic. There is one web site in the app pool that
receives some pretty heavy usage (40-50k visitors a day). However, the
server has specs that I'm sure should be able to handle it, and was handling
it well until a couple of months ago. Ever since then, we've had nothing but
problems. The app pool hangs frequently - typically around every 20 minutes
or so, if that.

As an initial attempt, I changed some settings in the app pool and in the
metabase file, but nothing seems to work quite well enough. I tried adding
an extra worker to the web garden, increasing thread request limits, etc. all
without success. My initial instinct was that the problem is ASP code
related, so I've spent quite a few days optimizing and fine tuning code
everywhere.

As a temporary "fix", I've enabled a frequent 5 minute worker recycle on the
app pool. This helps a lot, but does not FIX the problem.

To diagnose the problem, I've looked in the event viewer and I've run
DebugDiag numerous times.

Some common errors/warnings I get in the event viewer are:
- A process serving application pool 'PSX Extreme' exceeded time limits
during shut down. The process id was '3892'. (I'm pretty sure this means
something went wrong with a recycle?)
- Faulting application w3wp.exe, version 6.0.3790.1830, faulting module
unknown, version 0.0.0.0, fault address 0x05c07dc8.

The following are typical errors I get in DebugDiag:
- The following threads in w3wp.exe__PSX
Extreme__PID__3892__Date__06_28_2008__Time_10_44_1 2PM__609__IIS_COM+ Hang
Dump.dmp are displaying a message box. The call to display the message box
originated from user32!NtUserWaitMessage+c.
The text of the message being displayed is:
( 11 )
1.47% of threads blocked
- A COM+ STA Activity Pileup has been detected in w3wp.exe__PSX
Extreme__PID__5600__Date__06_28_2008__Time_10_44_1 7PM__609__IIS_COM+ Hang
Dump.dmp. There is at least one activity bound to every COM+ STA ThreadPool
thread, with the following threads having more than one activity bound:
20 21 22 23 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 44 45 46 47 48
49 50
See the COM+ STA ThreadPool Report for more detail.

Related Information:
-IIS6
-ASP
-MS SQL Server
-2 processors
-4gb RAM

I've already done TONS of reading on issues related to this matter. I've
search Google for related information to all of my errors and warnings, how
to analyze an IIS hang, read lots of articles by David Wang, but everything I
try, I just can't seem to get it to work.

If anyone has ANY words of wisdom or can interpret the latest DebugDiag
report on my app pool hang, I would forever be in your debt. Thanks for
taking the time to read this.

http://www.xoise.com/CrashHangReport.mht

What is vfbasics? Is that some custom .dll? I'd suggest that perhaps you
have deadlocked threads or something.

Also, you appear to have a lot of files named .inc. Is that extension mapped
to ASP (or similar)? Otherwise anyone can just type in
http://yoursite/yourIncfile.inc and get the contents of the inc file.

Cheers
Ken


"Dan" <Dan@discussions.microsoft.com> wrote in message
news:BC36F8E0-B8B1-49D4-87AC-4750495B26E3@microsoft.com...
> I'm the server administrator for a small business and I have one app pool
> that is very problematic. There is one web site in the app pool that
> receives some pretty heavy usage (40-50k visitors a day). However, the
> server has specs that I'm sure should be able to handle it, and was
> handling
> it well until a couple of months ago. Ever since then, we've had nothing
> but
> problems. The app pool hangs frequently - typically around every 20
> minutes
> or so, if that.
>
> As an initial attempt, I changed some settings in the app pool and in the
> metabase file, but nothing seems to work quite well enough. I tried
> adding
> an extra worker to the web garden, increasing thread request limits, etc.
> all
> without success. My initial instinct was that the problem is ASP code
> related, so I've spent quite a few days optimizing and fine tuning code
> everywhere.
>
> As a temporary "fix", I've enabled a frequent 5 minute worker recycle on
> the
> app pool. This helps a lot, but does not FIX the problem.
>
> To diagnose the problem, I've looked in the event viewer and I've run
> DebugDiag numerous times.
>
> Some common errors/warnings I get in the event viewer are:
> - A process serving application pool 'PSX Extreme' exceeded time limits
> during shut down. The process id was '3892'. (I'm pretty sure this means
> something went wrong with a recycle?)
> - Faulting application w3wp.exe, version 6.0.3790.1830, faulting module
> unknown, version 0.0.0.0, fault address 0x05c07dc8.
>
> The following are typical errors I get in DebugDiag:
> - The following threads in w3wp.exe__PSX
> Extreme__PID__3892__Date__06_28_2008__Time_10_44_1 2PM__609__IIS_COM+ Hang
> Dump.dmp are displaying a message box. The call to display the message box
> originated from user32!NtUserWaitMessage+c.
> The text of the message being displayed is:
> ( 11 )
> 1.47% of threads blocked
> - A COM+ STA Activity Pileup has been detected in w3wp.exe__PSX
> Extreme__PID__5600__Date__06_28_2008__Time_10_44_1 7PM__609__IIS_COM+ Hang
> Dump.dmp. There is at least one activity bound to every COM+ STA
> ThreadPool
> thread, with the following threads having more than one activity bound:
> 20 21 22 23 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 44 45 46 47 48
> 49 50
> See the COM+ STA ThreadPool Report for more detail.
>
> Related Information:
> -IIS6
> -ASP
> -MS SQL Server
> -2 processors
> -4gb RAM
>
> I've already done TONS of reading on issues related to this matter. I've
> search Google for related information to all of my errors and warnings,
> how
> to analyze an IIS hang, read lots of articles by David Wang, but
> everything I
> try, I just can't seem to get it to work.
>
> If anyone has ANY words of wisdom or can interpret the latest DebugDiag
> report on my app pool hang, I would forever be in your debt. Thanks for
> taking the time to read this.
>
> http://www.xoise.com/CrashHangReport.mht

Ken wrote on Mon, 30 Jun 2008 19:03:09 +1000:

> Also, you appear to have a lot of files named .inc. Is that extension
> mapped to ASP (or similar)? Otherwise anyone can just type in http://yoursite/yourIncfile.inc
> and get the contents of the inc file.

Not in IIS6 is doesn't - there isn't a default MIME mapping for .inc so IIS6
will return a 404 when requesting that extension unless you manually add it
as an allowed extension.

--
Dan

Thanks for your reply.

I don't know what vfbasics is actually. I noticed that a lot in the log as
well. I guess I need to try and figure that out. Are there any other things
you figure I need to look for?

But yeah, we use the .inc files as ASP includes in our CMS. They're
unaccessible frm the internet.

"Ken Schaefer" wrote:

> What is vfbasics? Is that some custom .dll? I'd suggest that perhaps you
> have deadlocked threads or something.
>
> Also, you appear to have a lot of files named .inc. Is that extension mapped
> to ASP (or similar)? Otherwise anyone can just type in
> http://yoursite/yourIncfile.inc and get the contents of the inc file.
>
> Cheers
> Ken
>
>
> "Dan" <Dan@discussions.microsoft.com> wrote in message
> news:BC36F8E0-B8B1-49D4-87AC-4750495B26E3@microsoft.com...
> > I'm the server administrator for a small business and I have one app pool
> > that is very problematic. There is one web site in the app pool that
> > receives some pretty heavy usage (40-50k visitors a day). However, the
> > server has specs that I'm sure should be able to handle it, and was
> > handling
> > it well until a couple of months ago. Ever since then, we've had nothing
> > but
> > problems. The app pool hangs frequently - typically around every 20
> > minutes
> > or so, if that.
> >
> > As an initial attempt, I changed some settings in the app pool and in the
> > metabase file, but nothing seems to work quite well enough. I tried
> > adding
> > an extra worker to the web garden, increasing thread request limits, etc.
> > all
> > without success. My initial instinct was that the problem is ASP code
> > related, so I've spent quite a few days optimizing and fine tuning code
> > everywhere.
> >
> > As a temporary "fix", I've enabled a frequent 5 minute worker recycle on
> > the
> > app pool. This helps a lot, but does not FIX the problem.
> >
> > To diagnose the problem, I've looked in the event viewer and I've run
> > DebugDiag numerous times.
> >
> > Some common errors/warnings I get in the event viewer are:
> > - A process serving application pool 'PSX Extreme' exceeded time limits
> > during shut down. The process id was '3892'. (I'm pretty sure this means
> > something went wrong with a recycle?)
> > - Faulting application w3wp.exe, version 6.0.3790.1830, faulting module
> > unknown, version 0.0.0.0, fault address 0x05c07dc8.
> >
> > The following are typical errors I get in DebugDiag:
> > - The following threads in w3wp.exe__PSX
> > Extreme__PID__3892__Date__06_28_2008__Time_10_44_1 2PM__609__IIS_COM+ Hang
> > Dump.dmp are displaying a message box. The call to display the message box
> > originated from user32!NtUserWaitMessage+c.
> > The text of the message being displayed is:
> > ( 11 )
> > 1.47% of threads blocked
> > - A COM+ STA Activity Pileup has been detected in w3wp.exe__PSX
> > Extreme__PID__5600__Date__06_28_2008__Time_10_44_1 7PM__609__IIS_COM+ Hang
> > Dump.dmp. There is at least one activity bound to every COM+ STA
> > ThreadPool
> > thread, with the following threads having more than one activity bound:
> > 20 21 22 23 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 44 45 46 47 48
> > 49 50
> > See the COM+ STA ThreadPool Report for more detail.
> >
> > Related Information:
> > -IIS6
> > -ASP
> > -MS SQL Server
> > -2 processors
> > -4gb RAM
> >
> > I've already done TONS of reading on issues related to this matter. I've
> > search Google for related information to all of my errors and warnings,
> > how
> > to analyze an IIS hang, read lots of articles by David Wang, but
> > everything I
> > try, I just can't seem to get it to work.
> >
> > If anyone has ANY words of wisdom or can interpret the latest DebugDiag
> > report on my app pool hang, I would forever be in your debt. Thanks for
> > taking the time to read this.
> >
> > http://www.xoise.com/CrashHangReport.mht
>
>

Actually, just did some research and found this:

vfbasics.dll is a Microsoft DLL with the description: "Application Verifier
Provider"

"Ken Schaefer" wrote:

> What is vfbasics? Is that some custom .dll? I'd suggest that perhaps you
> have deadlocked threads or something.
>
> Also, you appear to have a lot of files named .inc. Is that extension mapped
> to ASP (or similar)? Otherwise anyone can just type in
> http://yoursite/yourIncfile.inc and get the contents of the inc file.
>
> Cheers
> Ken
>
>
> "Dan" <Dan@discussions.microsoft.com> wrote in message
> news:BC36F8E0-B8B1-49D4-87AC-4750495B26E3@microsoft.com...
> > I'm the server administrator for a small business and I have one app pool
> > that is very problematic. There is one web site in the app pool that
> > receives some pretty heavy usage (40-50k visitors a day). However, the
> > server has specs that I'm sure should be able to handle it, and was
> > handling
> > it well until a couple of months ago. Ever since then, we've had nothing
> > but
> > problems. The app pool hangs frequently - typically around every 20
> > minutes
> > or so, if that.
> >
> > As an initial attempt, I changed some settings in the app pool and in the
> > metabase file, but nothing seems to work quite well enough. I tried
> > adding
> > an extra worker to the web garden, increasing thread request limits, etc.
> > all
> > without success. My initial instinct was that the problem is ASP code
> > related, so I've spent quite a few days optimizing and fine tuning code
> > everywhere.
> >
> > As a temporary "fix", I've enabled a frequent 5 minute worker recycle on
> > the
> > app pool. This helps a lot, but does not FIX the problem.
> >
> > To diagnose the problem, I've looked in the event viewer and I've run
> > DebugDiag numerous times.
> >
> > Some common errors/warnings I get in the event viewer are:
> > - A process serving application pool 'PSX Extreme' exceeded time limits
> > during shut down. The process id was '3892'. (I'm pretty sure this means
> > something went wrong with a recycle?)
> > - Faulting application w3wp.exe, version 6.0.3790.1830, faulting module
> > unknown, version 0.0.0.0, fault address 0x05c07dc8.
> >
> > The following are typical errors I get in DebugDiag:
> > - The following threads in w3wp.exe__PSX
> > Extreme__PID__3892__Date__06_28_2008__Time_10_44_1 2PM__609__IIS_COM+ Hang
> > Dump.dmp are displaying a message box. The call to display the message box
> > originated from user32!NtUserWaitMessage+c.
> > The text of the message being displayed is:
> > ( 11 )
> > 1.47% of threads blocked
> > - A COM+ STA Activity Pileup has been detected in w3wp.exe__PSX
> > Extreme__PID__5600__Date__06_28_2008__Time_10_44_1 7PM__609__IIS_COM+ Hang
> > Dump.dmp. There is at least one activity bound to every COM+ STA
> > ThreadPool
> > thread, with the following threads having more than one activity bound:
> > 20 21 22 23 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 44 45 46 47 48
> > 49 50
> > See the COM+ STA ThreadPool Report for more detail.
> >
> > Related Information:
> > -IIS6
> > -ASP
> > -MS SQL Server
> > -2 processors
> > -4gb RAM
> >
> > I've already done TONS of reading on issues related to this matter. I've
> > search Google for related information to all of my errors and warnings,
> > how
> > to analyze an IIS hang, read lots of articles by David Wang, but
> > everything I
> > try, I just can't seem to get it to work.
> >
> > If anyone has ANY words of wisdom or can interpret the latest DebugDiag
> > report on my app pool hang, I would forever be in your debt. Thanks for
> > taking the time to read this.
> >
> > http://www.xoise.com/CrashHangReport.mht
>
>

Dan wrote:
> I'm the server administrator for a small business and I have one app pool
> that is very problematic. There is one web site in the app pool that
> receives some pretty heavy usage (40-50k visitors a day). However, the
> server has specs that I'm sure should be able to handle it, and was handling
> it well until a couple of months ago. Ever since then, we've had nothing but
> problems. The app pool hangs frequently - typically around every 20 minutes
> or so, if that.
> <SNIP>
> http://www.xoise.com/CrashHangReport.mht

Your server seems to have ISAPI_Rewrite installed, and the thread
running it is displaying an interactive dialog box (perhaps an error)
and waiting for a response. You might see the dialog if you log in to
the server console session but then again you might not.

It *might* be this problem that is causing the rest of the ASP threads
to pile up waiting for ISAPI_Rewrite to respond.

If you need this particular filter, see if there is a newer version than
the one you currently have.

Alternatively I could be completely wrong :-) It's worth looking at anyway.

Cheers,

--
Chris M.

Remove pants to email me.

You have at least two different problems on this server.


First, you have Application Verifier activated for at least the
w3wp.exe process on this server. Please do not run it continuously in
production.

Application Verifier is very useful for identifying a large number of
common programming mistakes, but like most debugging tools, it needs
to be used sparingly and not be left running all the time in
production. For example, it can track down memory, handle, lock leaks,
COM issues, deadlocks, etc... but some of those checks cost a great
deal of CPU/memory to perform -- which can alter performance and
capacity characteristics of the server.

This alteration is marginally ok if you intentionally use specific
features to try and track down a pesky issue -- but it is NOT ok to
leave running in general.


Second, what objects are your ASP pages using. Please detail the
technology used for every single one of them. One (or more) of them
are STA (for example, if they wrap some VB6 object), and if you put
such objects into application scope, they can cause ASP pages in the
entire application pool to become single threaded and thus highly
unlikely to handle substantial load.

What is worse in your case is that some of the STA threads are blocked
behind a UI message box, likely detailing some error, and if every STA
thread (used by the ASP pages) is waiting on a lock held by this
thread... you will find your application quickly ground to a halt, no
matter if it is running on the fastest computer hardware on the
planet.

ASP in IIS6 is MTA by default, so the problem is likely introduced by
the application itself.


<soapbox>
Here are my thoughts on performance issues and interactions with IIS6
configuration. Only certain user application flaws can be temporarily
addressed with IIS6 configuration -- but the latent issue remains and
has to be fixed in the user code itself, before its usage grows beyond
the abilities of temporary bandaids.

i.e. Whenever you hear an application needs to "add more threads or
worker processes" to run successfully, it almost always indicate a
performance or architectural problem with that application being
temporarily masked by the configuration. Properly performant
applications on Windows need very few threads and run most
performantly within a single procress.

http://blogs.msdn.com/david.wang/arc...lications.aspx

http://blogs.msdn.com/david.wang/arc...f_threads.aspx

</soapbox>


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//




On Jun 30, 6:47*am, Dan <D...@discussions.microsoft.com> wrote:
> Actually, just did some research and found this:
>
> vfbasics.dll is a Microsoft DLL with the description: "Application Verifier
> Provider"
>
>
>
> "Ken Schaefer" wrote:
> > What is vfbasics? Is that some custom .dll? I'd suggest that perhaps you
> > have deadlocked threads or something.
>
> > Also, you appear to have a lot of files named .inc. Is that extension mapped
> > to ASP (or similar)? Otherwise anyone can just type in
> >http://yoursite/yourIncfile.incand get the contents of the inc file.
>
> > Cheers
> > Ken
>
> > "Dan" <D...@discussions.microsoft.com> wrote in message
> >news:BC36F8E0-B8B1-49D4-87AC-4750495B26E3@microsoft.com...
> > > I'm the server administrator for a small business and I have one app pool
> > > that is very problematic. *There is one web site in the app pool that
> > > receives some pretty heavy usage (40-50k visitors a day). *However, the
> > > server has specs that I'm sure should be able to handle it, and was
> > > handling
> > > it well until a couple of months ago. *Ever since then, we've had nothing
> > > but
> > > problems. *The app pool hangs frequently - typically around every 20
> > > minutes
> > > or so, if that.
>
> > > As an initial attempt, I changed some settings in the app pool and in the
> > > metabase file, but nothing seems to work quite well enough. *I tried
> > > adding
> > > an extra worker to the web garden, increasing thread request limits, etc.
> > > all
> > > without success. *My initial instinct was that the problem is ASP code
> > > related, so I've spent quite a few days optimizing and fine tuning code
> > > everywhere.
>
> > > As a temporary "fix", I've enabled a frequent 5 minute worker recycle on
> > > the
> > > app pool. *This helps a lot, but does not FIX the problem.
>
> > > To diagnose the problem, I've looked in the event viewer and I've run
> > > DebugDiag numerous times.
>
> > > Some common errors/warnings I get in the event viewer are:
> > > - A process serving application pool 'PSX Extreme' exceeded time limits
> > > during shut down. The process id was '3892'. (I'm pretty sure this means
> > > something went wrong with a recycle?)
> > > - Faulting application w3wp.exe, version 6.0.3790.1830, faulting module
> > > unknown, version 0.0.0.0, fault address 0x05c07dc8.
>
> > > The following are typical errors I get in DebugDiag:
> > > - The following threads in w3wp.exe__PSX
> > > Extreme__PID__3892__Date__06_28_2008__Time_10_44_1 2PM__609__IIS_COM+ Hang
> > > Dump.dmp are displaying a message box. The call to display the messagebox
> > > originated from user32!NtUserWaitMessage+c.
> > > The text of the message being displayed is:
> > > ( 11 )
> > > 1.47% of threads blocked
> > > - A COM+ STA Activity Pileup has been detected in w3wp.exe__PSX
> > > Extreme__PID__5600__Date__06_28_2008__Time_10_44_1 7PM__609__IIS_COM+ Hang
> > > Dump.dmp. There is at least one activity bound to every COM+ STA
> > > ThreadPool
> > > thread, with the following threads having more than one activity bound:
> > > 20 21 22 23 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 44 45 46 47 48
> > > 49 50
> > > See the COM+ STA ThreadPool Report for more detail.
>
> > > Related Information:
> > > -IIS6
> > > -ASP
> > > -MS SQL Server
> > > -2 processors
> > > -4gb RAM
>
> > > I've already done TONS of reading on issues related to this matter. *I've
> > > search Google for related information to all of my errors and warnings,
> > > how
> > > to analyze an IIS hang, read lots of articles by David Wang, but
> > > everything I
> > > try, I just can't seem to get it to work.
>
> > > If anyone has ANY words of wisdom or can interpret the latest DebugDiag
> > > report on my app pool hang, I would forever be in your debt. *Thanksfor
> > > taking the time to read this.
>
> > >http://www.xoise.com/CrashHangReport.mht- Hide quoted text -
>
> - Show quoted text -

Thanks for your reply. I've actually already removed the Application
Verifier. I updated the DebugDaig report here:
http://www.xoise.com/CrashHangReport.mht

The components I know of that I am running:
- ISAPI_Rewrite
- ASPImage
- A custom DLL created my the old programmer

I'm unaware of the technology behind these - is there some way to find out?
Also, how can I find out if my threads are blocked behind a UI message box?

Thanks,
Dan

"David Wang" wrote:

> You have at least two different problems on this server.
>
>
> First, you have Application Verifier activated for at least the
> w3wp.exe process on this server. Please do not run it continuously in
> production.
>
> Application Verifier is very useful for identifying a large number of
> common programming mistakes, but like most debugging tools, it needs
> to be used sparingly and not be left running all the time in
> production. For example, it can track down memory, handle, lock leaks,
> COM issues, deadlocks, etc... but some of those checks cost a great
> deal of CPU/memory to perform -- which can alter performance and
> capacity characteristics of the server.
>
> This alteration is marginally ok if you intentionally use specific
> features to try and track down a pesky issue -- but it is NOT ok to
> leave running in general.
>
>
> Second, what objects are your ASP pages using. Please detail the
> technology used for every single one of them. One (or more) of them
> are STA (for example, if they wrap some VB6 object), and if you put
> such objects into application scope, they can cause ASP pages in the
> entire application pool to become single threaded and thus highly
> unlikely to handle substantial load.
>
> What is worse in your case is that some of the STA threads are blocked
> behind a UI message box, likely detailing some error, and if every STA
> thread (used by the ASP pages) is waiting on a lock held by this
> thread... you will find your application quickly ground to a halt, no
> matter if it is running on the fastest computer hardware on the
> planet.
>
> ASP in IIS6 is MTA by default, so the problem is likely introduced by
> the application itself.
>
>
> <soapbox>
> Here are my thoughts on performance issues and interactions with IIS6
> configuration. Only certain user application flaws can be temporarily
> addressed with IIS6 configuration -- but the latent issue remains and
> has to be fixed in the user code itself, before its usage grows beyond
> the abilities of temporary bandaids.
>
> i.e. Whenever you hear an application needs to "add more threads or
> worker processes" to run successfully, it almost always indicate a
> performance or architectural problem with that application being
> temporarily masked by the configuration. Properly performant
> applications on Windows need very few threads and run most
> performantly within a single procress.
>
> http://blogs.msdn.com/david.wang/arc...lications.aspx
>
> http://blogs.msdn.com/david.wang/arc...f_threads.aspx
>
> </soapbox>
>
>
> //David
> http://w3-4u.blogspot.com
> http://blogs.msdn.com/David.Wang
> //
>
>
>
>
> On Jun 30, 6:47 am, Dan <D...@discussions.microsoft.com> wrote:
> > Actually, just did some research and found this:
> >
> > vfbasics.dll is a Microsoft DLL with the description: "Application Verifier
> > Provider"
> >
> >
> >
> > "Ken Schaefer" wrote:
> > > What is vfbasics? Is that some custom .dll? I'd suggest that perhaps you
> > > have deadlocked threads or something.
> >
> > > Also, you appear to have a lot of files named .inc. Is that extension mapped
> > > to ASP (or similar)? Otherwise anyone can just type in
> > >http://yoursite/yourIncfile.incand get the contents of the inc file.
> >
> > > Cheers
> > > Ken
> >
> > > "Dan" <D...@discussions.microsoft.com> wrote in message
> > >news:BC36F8E0-B8B1-49D4-87AC-4750495B26E3@microsoft.com...
> > > > I'm the server administrator for a small business and I have one app pool
> > > > that is very problematic. There is one web site in the app pool that
> > > > receives some pretty heavy usage (40-50k visitors a day). However, the
> > > > server has specs that I'm sure should be able to handle it, and was
> > > > handling
> > > > it well until a couple of months ago. Ever since then, we've had nothing
> > > > but
> > > > problems. The app pool hangs frequently - typically around every 20
> > > > minutes
> > > > or so, if that.
> >
> > > > As an initial attempt, I changed some settings in the app pool and in the
> > > > metabase file, but nothing seems to work quite well enough. I tried
> > > > adding
> > > > an extra worker to the web garden, increasing thread request limits, etc.
> > > > all
> > > > without success. My initial instinct was that the problem is ASP code
> > > > related, so I've spent quite a few days optimizing and fine tuning code
> > > > everywhere.
> >
> > > > As a temporary "fix", I've enabled a frequent 5 minute worker recycle on
> > > > the
> > > > app pool. This helps a lot, but does not FIX the problem.
> >
> > > > To diagnose the problem, I've looked in the event viewer and I've run
> > > > DebugDiag numerous times.
> >
> > > > Some common errors/warnings I get in the event viewer are:
> > > > - A process serving application pool 'PSX Extreme' exceeded time limits
> > > > during shut down. The process id was '3892'. (I'm pretty sure this means
> > > > something went wrong with a recycle?)
> > > > - Faulting application w3wp.exe, version 6.0.3790.1830, faulting module
> > > > unknown, version 0.0.0.0, fault address 0x05c07dc8.
> >
> > > > The following are typical errors I get in DebugDiag:
> > > > - The following threads in w3wp.exe__PSX
> > > > Extreme__PID__3892__Date__06_28_2008__Time_10_44_1 2PM__609__IIS_COM+ Hang
> > > > Dump.dmp are displaying a message box. The call to display the message box
> > > > originated from user32!NtUserWaitMessage+c.
> > > > The text of the message being displayed is:
> > > > ( 11 )
> > > > 1.47% of threads blocked
> > > > - A COM+ STA Activity Pileup has been detected in w3wp.exe__PSX
> > > > Extreme__PID__5600__Date__06_28_2008__Time_10_44_1 7PM__609__IIS_COM+ Hang
> > > > Dump.dmp. There is at least one activity bound to every COM+ STA
> > > > ThreadPool
> > > > thread, with the following threads having more than one activity bound:
> > > > 20 21 22 23 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 44 45 46 47 48
> > > > 49 50
> > > > See the COM+ STA ThreadPool Report for more detail.
> >
> > > > Related Information:
> > > > -IIS6
> > > > -ASP
> > > > -MS SQL Server
> > > > -2 processors
> > > > -4gb RAM
> >
> > > > I've already done TONS of reading on issues related to this matter. I've
> > > > search Google for related information to all of my errors and warnings,
> > > > how
> > > > to analyze an IIS hang, read lots of articles by David Wang, but
> > > > everything I
> > > > try, I just can't seem to get it to work.
> >
> > > > If anyone has ANY words of wisdom or can interpret the latest DebugDiag
> > > > report on my app pool hang, I would forever be in your debt. Thanks for
> > > > taking the time to read this.
> >
> > > >http://www.xoise.com/CrashHangReport.mht- Hide quoted text -
> >
> > - Show quoted text -
>

Also, I've read quite a few of your articles.

Around the time of the start of our problems, we noticed that attempting to
restart IIS never works on the first time. The application pools don't start
correctly until you restart a second time. Could this issue be related?

Thanks for your help,
Dan

"David Wang" wrote:

> You have at least two different problems on this server.
>
>
> First, you have Application Verifier activated for at least the
> w3wp.exe process on this server. Please do not run it continuously in
> production.
>
> Application Verifier is very useful for identifying a large number of
> common programming mistakes, but like most debugging tools, it needs
> to be used sparingly and not be left running all the time in
> production. For example, it can track down memory, handle, lock leaks,
> COM issues, deadlocks, etc... but some of those checks cost a great
> deal of CPU/memory to perform -- which can alter performance and
> capacity characteristics of the server.
>
> This alteration is marginally ok if you intentionally use specific
> features to try and track down a pesky issue -- but it is NOT ok to
> leave running in general.
>
>
> Second, what objects are your ASP pages using. Please detail the
> technology used for every single one of them. One (or more) of them
> are STA (for example, if they wrap some VB6 object), and if you put
> such objects into application scope, they can cause ASP pages in the
> entire application pool to become single threaded and thus highly
> unlikely to handle substantial load.
>
> What is worse in your case is that some of the STA threads are blocked
> behind a UI message box, likely detailing some error, and if every STA
> thread (used by the ASP pages) is waiting on a lock held by this
> thread... you will find your application quickly ground to a halt, no
> matter if it is running on the fastest computer hardware on the
> planet.
>
> ASP in IIS6 is MTA by default, so the problem is likely introduced by
> the application itself.
>
>
> <soapbox>
> Here are my thoughts on performance issues and interactions with IIS6
> configuration. Only certain user application flaws can be temporarily
> addressed with IIS6 configuration -- but the latent issue remains and
> has to be fixed in the user code itself, before its usage grows beyond
> the abilities of temporary bandaids.
>
> i.e. Whenever you hear an application needs to "add more threads or
> worker processes" to run successfully, it almost always indicate a
> performance or architectural problem with that application being
> temporarily masked by the configuration. Properly performant
> applications on Windows need very few threads and run most
> performantly within a single procress.
>
> http://blogs.msdn.com/david.wang/arc...lications.aspx
>
> http://blogs.msdn.com/david.wang/arc...f_threads.aspx
>
> </soapbox>
>
>
> //David
> http://w3-4u.blogspot.com
> http://blogs.msdn.com/David.Wang
> //
>
>
>
>
> On Jun 30, 6:47 am, Dan <D...@discussions.microsoft.com> wrote:
> > Actually, just did some research and found this:
> >
> > vfbasics.dll is a Microsoft DLL with the description: "Application Verifier
> > Provider"
> >
> >
> >
> > "Ken Schaefer" wrote:
> > > What is vfbasics? Is that some custom .dll? I'd suggest that perhaps you
> > > have deadlocked threads or something.
> >
> > > Also, you appear to have a lot of files named .inc. Is that extension mapped
> > > to ASP (or similar)? Otherwise anyone can just type in
> > >http://yoursite/yourIncfile.incand get the contents of the inc file.
> >
> > > Cheers
> > > Ken
> >
> > > "Dan" <D...@discussions.microsoft.com> wrote in message
> > >news:BC36F8E0-B8B1-49D4-87AC-4750495B26E3@microsoft.com...
> > > > I'm the server administrator for a small business and I have one app pool
> > > > that is very problematic. There is one web site in the app pool that
> > > > receives some pretty heavy usage (40-50k visitors a day). However, the
> > > > server has specs that I'm sure should be able to handle it, and was
> > > > handling
> > > > it well until a couple of months ago. Ever since then, we've had nothing
> > > > but
> > > > problems. The app pool hangs frequently - typically around every 20
> > > > minutes
> > > > or so, if that.
> >
> > > > As an initial attempt, I changed some settings in the app pool and in the
> > > > metabase file, but nothing seems to work quite well enough. I tried
> > > > adding
> > > > an extra worker to the web garden, increasing thread request limits, etc.
> > > > all
> > > > without success. My initial instinct was that the problem is ASP code
> > > > related, so I've spent quite a few days optimizing and fine tuning code
> > > > everywhere.
> >
> > > > As a temporary "fix", I've enabled a frequent 5 minute worker recycle on
> > > > the
> > > > app pool. This helps a lot, but does not FIX the problem.
> >
> > > > To diagnose the problem, I've looked in the event viewer and I've run
> > > > DebugDiag numerous times.
> >
> > > > Some common errors/warnings I get in the event viewer are:
> > > > - A process serving application pool 'PSX Extreme' exceeded time limits
> > > > during shut down. The process id was '3892'. (I'm pretty sure this means
> > > > something went wrong with a recycle?)
> > > > - Faulting application w3wp.exe, version 6.0.3790.1830, faulting module
> > > > unknown, version 0.0.0.0, fault address 0x05c07dc8.
> >
> > > > The following are typical errors I get in DebugDiag:
> > > > - The following threads in w3wp.exe__PSX
> > > > Extreme__PID__3892__Date__06_28_2008__Time_10_44_1 2PM__609__IIS_COM+ Hang
> > > > Dump.dmp are displaying a message box. The call to display the message box
> > > > originated from user32!NtUserWaitMessage+c.
> > > > The text of the message being displayed is:
> > > > ( 11 )
> > > > 1.47% of threads blocked
> > > > - A COM+ STA Activity Pileup has been detected in w3wp.exe__PSX
> > > > Extreme__PID__5600__Date__06_28_2008__Time_10_44_1 7PM__609__IIS_COM+ Hang
> > > > Dump.dmp. There is at least one activity bound to every COM+ STA
> > > > ThreadPool
> > > > thread, with the following threads having more than one activity bound:
> > > > 20 21 22 23 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 44 45 46 47 48
> > > > 49 50
> > > > See the COM+ STA ThreadPool Report for more detail.
> >
> > > > Related Information:
> > > > -IIS6
> > > > -ASP
> > > > -MS SQL Server
> > > > -2 processors
> > > > -4gb RAM
> >
> > > > I've already done TONS of reading on issues related to this matter. I've
> > > > search Google for related information to all of my errors and warnings,
> > > > how
> > > > to analyze an IIS hang, read lots of articles by David Wang, but
> > > > everything I
> > > > try, I just can't seem to get it to work.
> >
> > > > If anyone has ANY words of wisdom or can interpret the latest DebugDiag
> > > > report on my app pool hang, I would forever be in your debt. Thanks for
> > > > taking the time to read this.
> >
> > > >http://www.xoise.com/CrashHangReport.mht- Hide quoted text -
> >
> > - Show quoted text -
>