Apache2/MP2/Parllel::ForkManager and Segmentation Faults

...lots of them.

First off, let me thank Torsten for the help understanding some of the
environment issues with forking.

I have created a class for running external programs that will be a
little bit smart and a little bit configurable to allow processes to be
run by system() calls directly, IPC::Run3, which uses system, or by
$r->spawn_proc_prog() under mod_perl and also it provides for setting
environment variables via Env::C or by using $r->subprocess_env->set(
$var => $vars->{$var} ) if running under Apache/mod_perl.

Ok, now I've got my programs forking with the proper environment, but
I'm experiencing numerous segfaults. Same request issued multiple times
and a good number of them result in segmentation faults. That would be
on the order of once every other request, where each request could fork
about 7 or 8 times.

We're using DBI here as well, and I understand that there were related
issues with pre 1.3 versions of DBI.

I instantiate the ForkManager, and get into the loop, print out a
message right before I do my '$pm->start and next;', and then I get
this:

[Tue Aug 26 13:56:37 2008] [notice] child pid 19991 exit signal
Segmentation fault (11)

From what I've been able to determine so far, this happens when forking
the first process after creating the ForkManager object.

I'm running:

RHE Linux
Apache/2.2.9 (Unix) DAV/2 mod_apreq2-20051231/2.6.0
mod_perl/2.0.4
Perl/v5.10.0
DBI 1.605

I've tried changing the numbers the prefork mpm to allow for more
Clients, more MaxSpareServers. My config isn't tuned at all as you can
see:

StartServers 5
MinSpareServers 5
MaxSpareServers 15
MaxClients 150
MaxRequestsPerChild 128

Any thoughts or questions?

Eric D. Berg
Lehman Brothers
> Fixed Income Research
> 745 7th Avenue, 15th floor
> New York, NY 10019
> Phone +1 212 526 8118
>
>
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.

On Tue, Aug 26, 2008 at 2:04 PM, Berg, Eric <eric.berg@lehman.com> wrote:
> We're using DBI here as well

That's going to make forking a lot more complicated. You'll need to
close your DBI handles before you fork, or set InactiveDestroy on
them. You need to open new ones in the forked process to do any DBI
work.

If you strip out the DBI stuff from the forked processes, does it
still segfault?

- Perrin

I'm using Apache:BI.

I stripped out calls to code that use DBI and it still segfaults.

I've gone into the code and made sure that any method that does a DBI
call also uses $dbh->disconnect;

Eric

> -----Original Message-----
> From: pharkins@gmail.com [mailtoharkins@gmail.com] On
> Behalf Of Perrin Harkins
> Sent: Tuesday, August 26, 2008 2:12 PM
> To: Berg, Eric
> Cc: modperl@perl.apache.org; dlux@kapu.hu
> Subject: Re: Apache2/MP2/Parllel::ForkManager and Segmentation Faults
>
> On Tue, Aug 26, 2008 at 2:04 PM, Berg, Eric
> <eric.berg@lehman.com> wrote:
> > We're using DBI here as well
>
> That's going to make forking a lot more complicated. You'll need to
> close your DBI handles before you fork, or set InactiveDestroy on
> them. You need to open new ones in the forked process to do any DBI
> work.
>
> If you strip out the DBI stuff from the forked processes, does it
> still segfault?
>
> - Perrin
>
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.

On Tue, Aug 26, 2008 at 2:28 PM, Berg, Eric <eric.berg@lehman.com> wrote:
> I'm using Apache:BI.
>
> I stripped out calls to code that use DBI and it still segfaults.
>
> I've gone into the code and made sure that any method that does a DBI
> call also uses $dbh->disconnect;

With Apache:BI, you actually need to call $dbh->SUPER::disconnect.
However, you would only need to do this right before forking, not on
every call.

- Perrin

Hey, Perrin.

My DBI calls are within the code that is executed in the loop. Not
before. Even so, I tried using $dbh->super::disconnect(), and that had
no apparent affect. Still segfaulting.

I'm feeling like it's not the DBI stuff. Where else can I look?

Eric

> -----Original Message-----
> From: pharkins@gmail.com [mailtoharkins@gmail.com] On
> Behalf Of Perrin Harkins
> Sent: Tuesday, August 26, 2008 2:35 PM
> To: Berg, Eric
> Cc: modperl@perl.apache.org; dlux@kapu.hu
> Subject: Re: Apache2/MP2/Parllel::ForkManager and Segmentation Faults
>
> On Tue, Aug 26, 2008 at 2:28 PM, Berg, Eric
> <eric.berg@lehman.com> wrote:
> > I'm using Apache:BI.
> >
> > I stripped out calls to code that use DBI and it still segfaults.
> >
> > I've gone into the code and made sure that any method that
> does a DBI
> > call also uses $dbh->disconnect;
>
> With Apache:BI, you actually need to call $dbh->SUPER::disconnect.
> However, you would only need to do this right before forking, not on
> every call.
>
> - Perrin
>
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.

On Tue, Aug 26, 2008 at 3:25 PM, Berg, Eric <eric.berg@lehman.com> wrote:
> I'm feeling like it's not the DBI stuff. Where else can I look?

Your next step is probably to get a backtrace of the segfault and
figure out where it crashed.

- Perrin

Ok. I appreciate your help here, Perrin.

mod_backtrace gave me this:

[Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace backtrace for sig 11
(thread "pid" 24456)
[Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace main() is at 8062fb8
/home/pointsrv/apache/config/dev/modules/mod_backtrace.so[0xa3fa43]
/home/erberg/public_html/work/point-www/aut(ap_run_fatal_exception+0x2e)
[0x8079ce2]
/home/erberg/public_html/work/point-www/aut[0x807b0a0]
/lib/tls/libpthread.so.0[0x5b9a98]
/home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_PerlIO_flush+0
x68)[0x116e648]
/home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_pp_fork+0x45)[
0x1149b55]
/home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_runops_standar
d+0x18)[0x1102fe8]
/home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_call_sv+0x49a)
[0x110106a]
/home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback+0x
180)[0x10764f0]
/home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback_ru
n_handlers+0x35a)[0x1076b3a]
/home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback_pe
r_dir+0x53)[0x1076e83]
/home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_response_ha
ndler_cgi+0x11b)[0x107354b]
/home/erberg/public_html/work/point-www/aut(ap_run_handler+0x32)[0x80735
6e]
/home/erberg/public_html/work/point-www/aut(ap_invoke_handler+0xad)[0x80
73939]
/home/erberg/public_html/work/point-www/aut(ap_process_request+0x18d)[0x
807e669]
/home/erberg/public_html/work/point-www/aut[0x807c145]
/home/erberg/public_html/work/point-www/aut(ap_run_process_connection+0x
32)[0x8078e12]
/home/erberg/public_html/work/point-www/aut[0x80822c9]
/home/erberg/public_html/work/point-www/aut[0x808249e]
/home/erberg/public_html/work/point-www/aut[0x808252c]
[Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace end of backtrace
[Wed Aug 27 18:29:30 2008] [notice] child pid 24456 exit signal
Segmentation fault (11), possible coredump in /tmp

BTW, no core file was created in /tmp.

What do you think?

Eric

> -----Original Message-----
> From: pharkins@gmail.com [mailtoharkins@gmail.com] On
> Behalf Of Perrin Harkins
> Sent: Wednesday, August 27, 2008 11:18 AM
> To: Berg, Eric
> Cc: modperl@perl.apache.org; dlux@kapu.hu
> Subject: Re: Apache2/MP2/Parllel::ForkManager and Segmentation Faults
>
> On Tue, Aug 26, 2008 at 3:25 PM, Berg, Eric
> <eric.berg@lehman.com> wrote:
> > I'm feeling like it's not the DBI stuff. Where else can I look?
>
> Your next step is probably to get a backtrace of the segfault and
> figure out where it crashed.
>
> - Perrin
>
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.

Does anyone have time to take a look at this? I sure would appreciate
being able to move this forward and find a solution.

Thanks much in advance.

Eric

> -----Original Message-----
> From: Berg, Eric
> Sent: Wednesday, August 27, 2008 6:33 PM
> To: 'Perrin Harkins'
> Cc: modperl@perl.apache.org
> Subject: RE: Apache2/MP2/Parllel::ForkManager and Segmentation Faults
>
> Ok. I appreciate your help here, Perrin.
>
> mod_backtrace gave me this:
>
> [Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace backtrace
> for sig 11 (thread "pid" 24456)
> [Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace main() is
> at 8062fb8
> /home/pointsrv/apache/config/dev/modules/mod_backtrace.so[0xa3fa43]
>
/home/erberg/public_html/work/point-www/aut(ap_run_fatal_exception+0x2e)
[0x8079ce2]
> /home/erberg/public_html/work/point-www/aut[0x807b0a0]
> /lib/tls/libpthread.so.0[0x5b9a98]
>
/home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_PerlIO_flush+0
x68)[0x116e648]
>
/home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_pp_fork+0x45)[
0x1149b55]
>
/home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_runops_standar
d+0x18)[0x1102fe8]
>
/home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_call_sv+0x49a)
[0x110106a]
>
/home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback+0x
180)[0x10764f0]
>
/home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback_ru
n_handlers+0x35a)[0x1076b3a]
>
/home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback_pe
r_dir+0x53)[0x1076e83]
>
/home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_response_ha
ndler_cgi+0x11b)[0x107354b]
>
/home/erberg/public_html/work/point-www/aut(ap_run_handler+0x32)[0x80735
6e]
>
/home/erberg/public_html/work/point-www/aut(ap_invoke_handler+0xad)[0x80
73939]
>
/home/erberg/public_html/work/point-www/aut(ap_process_request+0x18d)[0x
807e669]
> /home/erberg/public_html/work/point-www/aut[0x807c145]
>
/home/erberg/public_html/work/point-www/aut(ap_run_process_connection+0x
32)[0x8078e12]
> /home/erberg/public_html/work/point-www/aut[0x80822c9]
> /home/erberg/public_html/work/point-www/aut[0x808249e]
> /home/erberg/public_html/work/point-www/aut[0x808252c]
> [Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace end of backtrace
> [Wed Aug 27 18:29:30 2008] [notice] child pid 24456 exit
> signal Segmentation fault (11), possible coredump in /tmp
>
> BTW, no core file was created in /tmp.
>
> What do you think?
>
> Eric
>
> > -----Original Message-----
> > From: pharkins@gmail.com [mailtoharkins@gmail.com] On
> > Behalf Of Perrin Harkins
> > Sent: Wednesday, August 27, 2008 11:18 AM
> > To: Berg, Eric
> > Cc: modperl@perl.apache.org; dlux@kapu.hu
> > Subject: Re: Apache2/MP2/Parllel::ForkManager and
> Segmentation Faults
> >
> > On Tue, Aug 26, 2008 at 3:25 PM, Berg, Eric
> > <eric.berg@lehman.com> wrote:
> > > I'm feeling like it's not the DBI stuff. Where else can I look?
> >
> > Your next step is probably to get a backtrace of the segfault and
> > figure out where it crashed.
> >
> > - Perrin
> >
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.