general performance issues

Hi.

This is not really an on-topic question, and is more of a general one.
But I figure that, for this type of "general", the combination of skills
of the people on this list might be just right to provide some pointers.
If it's too off-topic nevertheless, just send me packing, I won't insist.

The "top" snapshot below shows one of our old clunkers, doing
essentially nothing for now. It's an average desktop PC from a few
years back, running Linux Debian Etch, with a venerable Pentium II as
processor, 512 Mb Ram and your average PC disks from back then. It is
so old that there is in fact a problem still finding the memory cards
that fit in it.
Yet most of the time it faithfully does its job, which is to be a
development and test machine for a small team of programmers.
But sometimes it doesn't, and becomes rather slow for a while.

The programs that run on it are fairly typical of our production
machines (which are much newer/bigger/better), which is why I ask. I
figure that considering the respectable age of this system and its
limited everything, anything that we would do to improve its generic
performance would be all the more effective if applied to the production
servers, right ? or wrong ? in a generic way I mean.
Also, I want to add that we do not really have a problem yet, I am just
trying to gather some general information and general impressions,maybe
collect some tricks or tools for the future. We can easily get another
more recent PC to replace that one. But maybe then it would be harder to
put it under stress and see where there is a bottleneck.

We thus run on these machines : Apache 2.x with mod_perl 2.x, Tomcat
5.5, and mod_jk 1.2.x to connect them. Under Tomcat run a couple of
very small servlets, and one that seems to be really huge compared to
the others. Under Apache directly, we have a series of application
components running as mod_perl modules or as cgi-bin scripts.
We always access our web applications through Apache.
Apart from Tomcat and Apache-based stuff, there are also a few other
"daemons" running which you can see. A couple of them (the next 2 after
Tomcat) are java daemons,
and they process (and parse) XML files all day. A couple of stand-alone
perl processes are also daemons, and also process XML data.
The two "star" processes at the bottom are parts of a special database
system, written in C.
The one "python" process running there is "denyhosts".

This "top" display is sorted by "memory" (one of the options of memory
in top, "n", I'm not quite sure what it corresponds to).
The first thing that strikes me, is the huge memory allocated to Tomcat
and the other Java processes. I wonder if this unavoidable, or if it is
just misconfiguration on my part.
The "huge Tomcat application" is unfortunately one that we just buy, we
can't peek in the code to see what's going on there. We just know it is
(also) processing some large quantity of XML data, mostly when it starts
up. That almost paralyses our server for 10 minutes or so when we
restart Tomcat.
The other question is, does the memory shown as used by the Apache
processes seem normal/abnormal to anyone ? This being a prefork Apache,
if it seems abnormal, what kind of thing should I look at ? Or is it so
normal that I should not even spend time on it ?
(I already load a fair number of modules in a startup script, but maybe
I should load even more of them there, if there is no penalty).

Thanks for any reactions, even if it is to tell me that I don't know
what I'm talking about or that this isn't the right place. But then, be
as kind maybe as pointing me to the right place.


Sample "top" :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6226 tomcat55 25 0 506m 188m 10m S 0.0 37.3 127:01.74 java
5937 root 15 0 267m 28m 8056 S 0.0 5.6 24:20.96 java
8613 root 24 0 260m 15m 9332 S 0.0 3.1 0:02.16 java
8640 root 15 0 29560 15m 5984 S 0.0 3.0 0:02.10 apache2
8632 efs 15 0 16964 12m 1968 S 0.0 2.6 0:06.96 MiraLoader.pl
8665 www-data 15 0 29692 12m 3016 S 0.0 2.4 0:08.82 apache2
8662 www-data 15 0 29692 12m 3016 S 0.0 2.4 0:08.74 apache2
12414 www-data 15 0 29560 10m 1624 S 0.0 2.2 0:08.32 apache2
12415 www-data 15 0 29560 10m 1624 S 0.0 2.2 0:08.33 apache2
18880 www-data 15 0 29560 10m 1624 S 0.0 2.2 0:04.84 apache2
18881 www-data 18 0 29560 10m 1624 S 0.0 2.2 0:04.37 apache2
18882 www-data 15 0 29560 10m 1624 S 0.0 2.2 0:04.55 apache2
8666 efs 15 0 13532 9724 2000 S 0.0 1.9 0:02.40 MiraFiler.pl
21408 root 18 0 9940 3884 1376 S 0.0 0.8 32:10.91 python
24934 efs 17 0 11708 2812 2172 S 0.0 0.5 0:00.06 smbd
23831 efs 18 0 14548 2680 1724 S 0.0 0.5 0:03.35 MiraASci.pl
23848 efs 15 0 11696 2568 1708 S 0.0 0.5 0:03.03 ifsser.pl
24752 root 15 0 7804 2324 1836 S 0.0 0.4 0:00.12 sshd
24763 root 15 0 7840 2292 1848 S 0.3 0.4 0:00.32 sshd
26537 root 16 0 11416 2104 1624 S 0.0 0.4 3:57.05 smbd
27888 star 15 0 6600 1968 1544 S 0.0 0.4 0:03.86 star
27886 star 16 0 5004 1840 1440 S 0.0 0.4 0:30.21 star

André

André Warnier wrote:
> If it's too off-topic nevertheless, just send me packing, I won't insist.

I might be alone in this but I think apache memory use is something all
heavy mod_perl users have to get to grips with, so I mentally
s/java|tomcat/mod_perl/g'd and read your very long email....
>
> But sometimes it doesn't, and becomes rather slow for a while.
Find out if it's *actual* slowness or just HTTP requests being queued up
because none of the apache children are free to service the request.
>
> anything that we would do to improve its generic performance would be
> all the more effective if applied to the production servers, right ?
> or wrong ? in a generic way I mean.
I agree but with one caveat - memory is very cheap at the moment.
Strike a good balance between spending hours trimming the memory
footprint of your app and buying more RAM.

> The first thing that strikes me, is the huge memory allocated to
> Tomcat and the other Java processes. I wonder if this unavoidable, or
> if it is just misconfiguration on my part.
> The "huge Tomcat application" is unfortunately one that we just buy,
> we can't peek in the code to see what's going on there. We just know
> it is (also) processing some large quantity of XML data, mostly when
> it starts up. That almost paralyses our server for 10 minutes or so
> when we restart Tomcat.
Tongue in cheek, thats what you get for not rewriting it in perl
Seriously though, in your own apps make sure you're not loading large
XML files fully into RAM before parsing them. Use a SAX or similar
parser to treat large XML files like a stream.
> The other question is, does the memory shown as used by the Apache
> processes seem normal/abnormal to anyone ? This being a prefork
> Apache, if it seems abnormal, what kind of thing should I look at ?
> Or is it so normal that I should not even spend time on it ?
Those are very lean looking apache's by my own experience, I don't think
you'll do much better.

hope that's useful,

John