Questions on mod_proxy Apache 2.0.52

[Jeffrey Burgoyne]

I'm looking for a bit of direction on a few issues related to mod_proxy
and the Apache 2.X series.

Currently I'm in charge of Apache on a large website running 1.3.26
apache. Over the past few years the department has had a proliferation of
technologies for back end servers. I have been using mod_proxy to manage
connectivity to all of the servers in question. Unfortunately in the last
year our server has had issues handling the load restraints and buggy
application servers. To alleviate the immediate issues I made customized
changes to the 1.3.26 Apache mod_proxy to handle various timeout
conditions and real time monitoring. We also have been given the funding
to deploy a robust front end web server machine (An IBM blade center).

Given the magnitude of the migration to the new machine, I was given the
go ahead to investigate Apache 2.X as a better solution for the front end
web server. Notably of importance to us was the fact that a proxy timeout
directive was built into 2.X, and we would not need to worry about the
custom code. The load balancing failover that is being talked about will
also be of significant interest to us as it becomes available. I have run
into three issues, however.

1) SSL proxying. Due to security policies, we have a number of back end
app servers that require SSL from the client to the server. Therefore SSL
based proxying is a requirement. I have never seen a definitive statement
as to whether SSL proxying is supported, but I've seen indications it is
not, and confirmed in my tests that it did not work. Is there any plans to
implement this feature?

2) Timeout Directive. I tried using this with the test suite that I used
for my mod_proxy changes, and did not get the intended results. For
example, I wrote a cgi that wais 30 seconds before passing back a
response, and set the timeout to 10 seconds. On my version the proxy would
give up after 10 seconds of no data transfer. This did not happen with the
2.X timeout. Can someone give a better explanation of what this timeout
handles and whether I possibly made a configuration mistake.

3) Monitoring. My proxy changes wrote out a customize log entry upon
failure. I then wrote a program which analyzed this log in real time and
sent out warnings on configurable intervals when configurable thresholds
were breached. Assuming I can get 1 and 2 sorted out, I'd be willing to
work on this third item as an enhancement to mod_proxy.


Thanks


Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc
***@keenuh.com

[Graham Leggett]

Post by Jeffrey Burgoyne
1) SSL proxying. Due to security policies, we have a number of back end
app servers that require SSL from the client to the server. Therefore SSL
based proxying is a requirement. I have never seen a definitive statement
as to whether SSL proxying is supported, but I've seen indications it is
not, and confirmed in my tests that it did not work. Is there any plans to
implement this feature?

If it doesn't work now, it is definitely desirable to make it work.

The v2.0 mod_proxy talks to the backend servers using the standard
filter stack, so making it talk SSL to the backend should be as
straightforward as adding the right filters to the stack at the right
time under the right circumstances.

If you're willing to submit code for this, I will definitely support
getting this into v2.0 (not only v2.2).

Post by Jeffrey Burgoyne
2) Timeout Directive. I tried using this with the test suite that I used
for my mod_proxy changes, and did not get the intended results. For
example, I wrote a cgi that wais 30 seconds before passing back a
response, and set the timeout to 10 seconds. On my version the proxy would
give up after 10 seconds of no data transfer. This did not happen with the
2.X timeout. Can someone give a better explanation of what this timeout
handles and whether I possibly made a configuration mistake.

As I recall, the timeout directive handles the timeout after a
connection has been established - this definitely would need to be
looked at if it's not working properly.

Post by Jeffrey Burgoyne
3) Monitoring. My proxy changes wrote out a customize log entry upon
failure. I then wrote a program which analyzed this log in real time and
sent out warnings on configurable intervals when configurable thresholds
were breached. Assuming I can get 1 and 2 sorted out, I'd be willing to
work on this third item as an enhancement to mod_proxy.

This is also something really useful - please submit the patch :)

Regards,
Graham
--

[Jeffrey Burgoyne]

Graham, thanks for the reply. See comments below.

Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc

Post by Graham Leggett

Post by Jeffrey Burgoyne
1) SSL proxying. Due to security policies, we have a number of back end
app servers that require SSL from the client to the server. Therefore SSL
based proxying is a requirement. I have never seen a definitive statement
as to whether SSL proxying is supported, but I've seen indications it is
not, and confirmed in my tests that it did not work. Is there any plans to
implement this feature?

If it doesn't work now, it is definitely desirable to make it work.
The v2.0 mod_proxy talks to the backend servers using the standard
filter stack, so making it talk SSL to the backend should be as
straightforward as adding the right filters to the stack at the right
time under the right circumstances.
If you're willing to submit code for this, I will definitely support
getting this into v2.0 (not only v2.2).

I may try to do this, but unfortunately due to time constraints I cannot
commit to working on this as a priority. From your explanation it does
look easier than I would have imagined, so I will try to take a cursory
look.

Post by Graham Leggett

Post by Jeffrey Burgoyne
2) Timeout Directive. I tried using this with the test suite that I used
for my mod_proxy changes, and did not get the intended results. For
example, I wrote a cgi that wais 30 seconds before passing back a
response, and set the timeout to 10 seconds. On my version the proxy would
give up after 10 seconds of no data transfer. This did not happen with the
2.X timeout. Can someone give a better explanation of what this timeout
handles and whether I possibly made a configuration mistake.

As I recall, the timeout directive handles the timeout after a
connection has been established - this definitely would need to be
looked at if it's not working properly.

Perhaps I configured it wrong. I have :

ProxyPass /cgi-bin/ http://strategis.ic.gc.ca/cgi-bin/
ProxyTimeout 10


When accessing /cgi-bin/timeoutin35seconds.cgi I immediately get the
connection as expected. The CGI though will not do anythign for 35 seconds
before sending back an HTML page. This did not timeout. Was I missing
something?

Post by Graham Leggett

Post by Jeffrey Burgoyne
3) Monitoring. My proxy changes wrote out a customize log entry upon
failure. I then wrote a program which analyzed this log in real time and
sent out warnings on configurable intervals when configurable thresholds
were breached. Assuming I can get 1 and 2 sorted out, I'd be willing to
work on this third item as an enhancement to mod_proxy.

This is also something really useful - please submit the patch :)

I looked at this and will pretty much have to start from scratch. The way
did the proxy timeout in 1.3.26 is too different change to make my change
work. I did it by creating my own timeout function which took a the
seconds for a timeout followed by the URL being accessed. I then stored
the back end server socket in a global and used the callback and alarm.
Something like this :

(In http_main.c)

static void proxy_timeout(int sig)
{
strat_cancel_proxy_connection((request_rec *) timeout_req);
timeout(sig);
}


API_EXPORT(void) ap_config_timeout(int timeout_period, char *name,
request_rec *r)
{
#ifdef NETWARE
get_tsd
#endif
timeout_req = r;
timeout_name = name;

if (timeout_period < 1)
{
timeout_period = r->server->timeout;
}
ap_set_callback_and_alarm(proxy_timeout, timeout_period);
}

And in mod_proxy.c

void
strat_cancel_proxy_connection(request_rec *the_request)
{
extern int proxy_server_socket;

strat_log_proxy_failure(the_request->unparsed_uri);

if (proxy_server_socket != 0)
{
close(proxy_server_socket);
}

the_request->status = HTTP_BAD_GATEWAY;
ap_proxyerror(the_request, HTTP_BAD_GATEWAY, "Upstream Server
Unavailable.");
ap_send_error_response(the_request, 0);

return;
}


Teh changes made for the 2.0 were using the socket timeout I believe,
which is a more generic timeout function with its own function called upon
timeout.


So I will take a look at the logging, but for the most part will not be
able to reuse what I have done. I willbe able to reuse the monitroing
program though, as it is totally outside of Apache.


Thanks again.

Jeff

Post by Graham Leggett
Regards,
Graham
--

[Jeffrey Burgoyne]

Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc

Post by Jeffrey Burgoyne
1) SSL proxying.

This is now working.

Post by Jeffrey Burgoyne
2) Timeout Directive.

This one is still confounded me. I added in some debugging statements into
the code. For the startup phase, the value is definitely set properly.
When the proxy request is made, however, the set_timeout seems to be set
to 0 and hence the server timeout is then used.

I'll keep looking at this one and try to figure out exactly where the
value gets blanked. If someone wants to verify I've set up :

strategis.ic.gc.ca/cgi-bin/utils/timeout.cgi

is set up to hold the connection open, but return no data. Be warned the
firewall on our side will forcefully close this connection after 40
seconds, so try a lower timeout value.

Post by Jeffrey Burgoyne
3) Monitoring.

I have a December/January timeframe for delivery of the project, so I'll
be looking at trying this sometime within the next few months as time
allows.

Thanks for the help everyone.

Jeff

[Jeffrey Burgoyne]

Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc

Post by Jeffrey Burgoyne

Post by Jeffrey Burgoyne
2) Timeout Directive.

This one is still confounded me. I added in some debugging statements into
the code. For the startup phase, the value is definitely set properly.
When the proxy request is made, however, the set_timeout seems to be set
to 0 and hence the server timeout is then used.

Interesting results. I've trapped all places were the timeout_set variable
is set, and it seems to be 0 upon startup (default), set to 1 when the
ProxyTimeout is made, set to 1 on the merge config, yet 0 in the
ap_proxy_connect_to_backend function.

I have verified that the value placed into the timeout itself
(conf->timeout) is indeed the value that was set when set_proxy_timeout is
called. Exactly how the timeout_set is getting set to 0 is the mystery.
I'll keep looking.

Jeff

[Eric J. Hansen]

Post by Jeffrey Burgoyne
1) SSL proxying. Due to security policies, we have a number
of back end app servers that require SSL from the client to
the server. Therefore SSL based proxying is a requirement. I
have never seen a definitive statement as to whether SSL
proxying is supported, but I've seen indications it is not,
and confirmed in my tests that it did not work. Is there any
plans to implement this feature?

We're using Apache2 mod_proxy as a reverse proxy with mod_ssl enabled,
proxying to a back-end WWW server over https. My understanding is
that the proxy is an SSL termination point, and it then opens a
new SSL connection to the back-end. You need to install certs on
both Apache and the back-end (although they can be the exact
same cert.) You also need to specify the "SSLProxyEngine On" directive
in your httpd.conf file.

A fact that you should also be aware of is that, technically speaking,
the request/response are being decrypted and re-encrypted in the
Apache mod_proxy process before being proxied onward to network.
Theoretically, this exposes you to man-in-the-middle issues... so good
host security and the latest patches are essential.

cheers
Eric

[Jeffrey Burgoyne]

Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc

Post by Eric J. Hansen

Post by Jeffrey Burgoyne
1) SSL proxying. Due to security policies, we have a number
of back end app servers that require SSL from the client to
the server. Therefore SSL based proxying is a requirement. I
have never seen a definitive statement as to whether SSL
proxying is supported, but I've seen indications it is not,
and confirmed in my tests that it did not work. Is there any
plans to implement this feature?

We're using Apache2 mod_proxy as a reverse proxy with mod_ssl enabled,
proxying to a back-end WWW server over https. My understanding is
that the proxy is an SSL termination point, and it then opens a
new SSL connection to the back-end. You need to install certs on
both Apache and the back-end (although they can be the exact
same cert.) You also need to specify the "SSLProxyEngine On" directive
in your httpd.conf file.

Thanks, I think that is it. Coming from the 1.3 world I simply used an
https in the url for the reverse.

When looking at the web site, I thought all the proxy directives were
under http://httpd.apache.org/docs-2.0/mod/mod_proxy.html . I missed the
comment on the third paragraph of the summary. Perhaps that section should
be a bit more prominent.

Post by Eric J. Hansen
A fact that you should also be aware of is that, technically speaking,
the request/response are being decrypted and re-encrypted in the
Apache mod_proxy process before being proxied onward to network.
Theoretically, this exposes you to man-in-the-middle issues... so good
host security and the latest patches are essential.

Absolutely. In fact, most people where I am on contract think security is
way overblown. We keep the whole environment very tightly locked down.

Post by Eric J. Hansen
cheers
Eric

[Cahya Wirawan]

Post by Jeffrey Burgoyne
1) SSL proxying. Due to security policies, we have a number of back end
app servers that require SSL from the client to the server. Therefore SSL
based proxying is a requirement. I have never seen a definitive statement
as to whether SSL proxying is supported, but I've seen indications it is
not, and confirmed in my tests that it did not work. Is there any plans to
implement this feature?

Hi,
We are using ssl mod_proxy since more than 2 years, you just need to use
ip based virtual host and the certificate in the reverse proxy. that's
all I think.

regards,
cahya.