Load peaks and throttling

Hi,

I'm currently working on a "large" (for some meaning of "large", of course)
community web site, using mp2 with mostly handlers and some registry
scripts.

The architecture is basically composed of:
- a few load-balanced frontend apaches
- several load-balanced mp2 backends
- a replicated mysql db

From time to time, we are seeing very high peaks of traffic on
our backends, with db connections and backend threads quickly going up.
This rapidly causes backend load to increase and general slowdown
of the application.

Most of the times, these peaks are caused by search engine bots
or automated spam clients.

I'm trying to fully understand this problem, but I'm considering
implementing throttling at the backend (maybe front?) level,
so I was wondering what is the current state-of-the-art
or best practice about this.

Any pointers or suggestions?

--
Cosimo

On Mon, Jul 21, 2008 at 6:29 PM, Cosimo Streppone <cosimo@streppone.it> wrote:
> I'm trying to fully understand this problem, but I'm considering
> implementing throttling at the backend (maybe front?) level,
> so I was wondering what is the current state-of-the-art
> or best practice about this.

If you search the apache modules database for "throttle" and
"bandwidth" you'll see some front-end options. For the backend, the
CPAN modules that do this are mostly mod_perl 1, but wouldn't be hard
to port.

- Perrin

In data 22 luglio 2008 alle ore 02:53:40, Perrin Harkins <perrin@elem.com>
ha scritto:

> On Mon, Jul 21, 2008 at 6:29 PM, Cosimo Streppone <cosimo@streppone.it>
> wrote:
>> implementing throttling at the backend (maybe front?) level,
>
> [...] For the backend, the CPAN modules that do this are mostly mod_perl
> 1

By CPAN modules you mean Apache::Throttle?

> but wouldn't be hard to port.

I can probably spend some time on that.

--
Cosimo