mod_proxy buffers chunked data... - Apache

* inder sabharwal <inder1@sbcglobal.net> [2005-11-15 11:43]:

> I have a server that sends data using Transfer-Encoding: Chunked, but
> mod_proxy is buffering up the data until it hits the internal limit of
> 8K. Our client application expects to receive data from the server over
> a period of time, and doesnot know the length of the data in advance.

> I have looked at bugs 19954 and 34876, which claim to have fixed this in
> the latest version of apache, but apparently it is not so. I have tried
> it on the following apache servers:

> 1.) Server version: Apache/2.0.46
> Server built: Nov 5 2004 10:58:21

> 2.) Server version: Apache/2.0.55
> Server built: Nov 15 2005 00:06:22

> 3.) The 2.1 beta available from apache.org.

> Is there a patch available somewhere that I could use or a configuration
> item that I may be missing out?

I am facing the same problem. I've tested against Apache/2.0.48.

I'm glad to see that you've tested against other versions, and can
confirm my experience. Unfortunately, I don't have an answer for how
to fix this. I'd like to hear one.

Is this considered a bug for a feature? I can see how it might be
benificial to buffer chunks and send them along as a larger chunk,
for some applications.

It might be a missuse of HTTP to expect chunking to work this way,
but it would certianly make it easier on my particular application.

--
Alan Gutierrez - alan@engrm.com - http://engrm.com/blogometer/

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

On 11/15/05, inder sabharwal <inder1@sbcglobal.net> wrote:
> Hi:
> I have a server that sends data using Transfer-Encoding: Chunked, but
> mod_proxy is buffering up the data until it hits the internal limit of
> 8K. Our client application expects to receive data from the server over
> a period of time, and doesnot know the length of the data in advance.

You can try adjusting ProxyIOBufferSize.

Joshua.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

* Alan Gutierrez <alan-httpd-users@engrm.com> [2005-11-17 11:42]:
> * inder sabharwal <inder1@sbcglobal.net> [2005-11-15 11:43]:
>
> > I have a server that sends data using Transfer-Encoding: Chunked, but
> > mod_proxy is buffering up the data until it hits the internal limit of
> > 8K. Our client application expects to receive data from the server over
> > a period of time, and doesnot know the length of the data in advance.
>
> > I have looked at bugs 19954 and 34876, which claim to have fixed this in
> > the latest version of apache, but apparently it is not so. I have tried
> > it on the following apache servers:
>
> > 1.) Server version: Apache/2.0.46
> > Server built: Nov 5 2004 10:58:21
>
> > 2.) Server version: Apache/2.0.55
> > Server built: Nov 15 2005 00:06:22
>
> > 3.) The 2.1 beta available from apache.org.
>
> > Is there a patch available somewhere that I could use or a configuration
> > item that I may be missing out?
>
> I am facing the same problem. I've tested against Apache/2.0.48.
>
> I'm glad to see that you've tested against other versions, and can
> confirm my experience. Unfortunately, I don't have an answer for how
> to fix this. I'd like to hear one.
>
> Is this considered a bug for a feature? I can see how it might be
> benificial to buffer chunks and send them along as a larger chunk,
> for some applications.
>
> It might be a missuse of HTTP to expect chunking to work this way,
> but it would certianly make it easier on my particular application.

It doesn't look like a misuse, since I've encountered a lot of
similiar complaints. So...

The solution that was applied to mod_proxy appears to perform some
sort of test to see if a read from the upstream server will block.
If that's the case, the buffer is flushed before 8192.

Is there something I can add to my Servlet that says, yes, you're
going to block?

--
Alan Gutierrez - alan@engrm.com - http://engrm.com/blogometer/

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

* Joshua Slive <jslive@gmail.com> [2005-11-17 14:15]:
> On 11/15/05, inder sabharwal <inder1@sbcglobal.net> wrote:
> > Hi:
> > I have a server that sends data using Transfer-Encoding: Chunked, but
> > mod_proxy is buffering up the data until it hits the internal limit of
> > 8K. Our client application expects to receive data from the server over
> > a period of time, and doesnot know the length of the data in advance.
>
> You can try adjusting ProxyIOBufferSize.

That doesn't work. The buffering isn't for the sake of caching, it's
to gather up enough data so that it is packet sized.

Setting ProxyIOBufferSize to 0 or 1 doesn't keep mod_proxy from
attempting to fill a packet.

--
Alan Gutierrez - alan@engrm.com - http://engrm.com/blogometer/

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

* Alan Gutierrez <alan-httpd-users@engrm.com> [2005-11-17 11:42]:
> * inder sabharwal <inder1@sbcglobal.net> [2005-11-15 11:43]:
>
> > I have a server that sends data using Transfer-Encoding: Chunked, but
> > mod_proxy is buffering up the data until it hits the internal limit of
> > 8K. Our client application expects to receive data from the server over
> > a period of time, and doesnot know the length of the data in advance.
>
> > I have looked at bugs 19954 and 34876, which claim to have fixed this in
> > the latest version of apache, but apparently it is not so. I have tried
> > it on the following apache servers:
>
> > 1.) Server version: Apache/2.0.46
> > Server built: Nov 5 2004 10:58:21
>
> > 2.) Server version: Apache/2.0.55
> > Server built: Nov 15 2005 00:06:22
>
> > 3.) The 2.1 beta available from apache.org.
>
> > Is there a patch available somewhere that I could use or a configuration
> > item that I may be missing out?
>
> I am facing the same problem. I've tested against Apache/2.0.48.
>
> I'm glad to see that you've tested against other versions, and can
> confirm my experience. Unfortunately, I don't have an answer for how
> to fix this. I'd like to hear one.
>
> Is this considered a bug for a feature? I can see how it might be
> benificial to buffer chunks and send them along as a larger chunk,
> for some applications.
>
> It might be a missuse of HTTP to expect chunking to work this way,
> but it would certianly make it easier on my particular application.

Any progress on this?

I'm getting ready to use a patched mod_proxy. Is there something
like mod_push or mod_tunnel?

--
Alan Gutierrez - alan@engrm.com - http://engrm.com/blogometer/

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

On Tue, Nov 15, 2005 at 08:42:54AM -0800, inder sabharwal wrote:
> I have a server that sends data using Transfer-Encoding: Chunked, but
> mod_proxy is buffering up the data until it hits the internal limit of
> 8K. Our client application expects to receive data from the server over
> a period of time, and doesnot know the length of the data in advance.
....
> 3.) The 2.1 beta available from apache.org.

This should be working as you expect out of the box in all recent 2.1.x
releases (but not in 2.0.x) unless it has been broken recently.

Can you try the 2.1.10 test tarball? There are a number of fixes for
the proxy there.

http://people.apache.org/~pquerna/dev/httpd-2.1.10/

joe

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org