"Secure connection truncated" error - Apache crash

[jbz]

Cheers all. I have a RHEL5 system using Subversion 1.4.5 compiled from source, being served by the distro's Apache (httpd-2.2.3-7.el5) over SSL using the distro's neon-0.25.5-5.1 at runtime AFAICT (although subversion was compiled with neon source in the tree, from subversion-deps). It is a fsfs repository on an ext3 filesystem (local disk) using mod_auth_kerb to auth to a windows DC. That is all working fine. However, while it seems to work OK for small test project trees (this is a new installation, natch, not yet in production) trying to import or checkout a large project tree will swiftly produce an error. The checkout errors look like this:

---cut---
svn: REPORT request failed on '/svn/repo/!svn/vcc/default'
svn: REPORT of '/svn/repo/!svn/vcc/default': Could not read response body: Secure connection truncated (https://svn.myco.com)
---cut---

...or like this, apparently random choice as to which:

---cut---
svn: REPORT request failed on '/svn/repo/!svn/vcc/default'
svn: REPORT of '/svn/repo/!svn/vcc/default': Could not read chunk delimiter: Secure connection truncated (https://svn.myco.com)
---cut---

I had taken the following steps to try to ameliorate the problem already:

- increased the timeout to 1000 seconds (which fixed an earlier connection issue)
- tried disabling apache worker mpm and relying on prefork, as per a suggestion for someone losing connections using svn+ssh. That didn't work; re-enabled.
- disabled http compression (by commenting out the loading of mod_deflate). Left that disabled just in case.
- enabled keepalive and upped the requests per connection. No change.


Then:

After investigating and watching apache while the checkout was running, I noted that the working apache process would grow during the checkout to consume appro. 27m of resident memory and then suddenly vanish when the error occured, briefly showing up in the process list as httpd <defunct>. Checking the apache main error logs (and why can't these show up in the vhost logs!?!? heh.) showed the following, which I verified to be happening when the checkout failures occurred:

---cut---
*** glibc detected *** /usr/sbin/httpd: corrupted double-linked list: 0x0b4ddba0 ***
======= Backtrace: =========
/lib/libc.so.6[0x6f12da]
/lib/libc.so.6[0x6f35d3]
/lib/libc.so.6(__libc_malloc+0x7e)[0x6f4dae]
/usr/lib/libkrb5.so.3[0x379a97]
/etc/httpd/modules/mod_auth_kerb.so[0x590209]
/usr/sbin/httpd(ap_run_check_user_id+0x6d)[0x4c985d]
/usr/sbin/httpd(ap_process_request_internal+0x207)[0x4cab57]
/usr/sbin/httpd(ap_sub_req_method_uri+0xcb)[0x4caf6b]
/usr/lib/httpd/modules/mod_dav_svn.so(dav_svn_authz_read+0x1c9)[0x537db9]
/usr/local/lib/libsvn_repos-1.so.0[0x9c5625]
/usr/local/lib/libsvn_repos-1.so.0[0x9c64fb]
/usr/local/lib/libsvn_repos-1.so.0[0x9c6f89]
/usr/local/lib/libsvn_repos-1.so.0[0x9c68c5]
/usr/local/lib/libsvn_repos-1.so.0[0x9c6f89]
/usr/local/lib/libsvn_repos-1.so.0[0x9c68c5]
/usr/local/lib/libsvn_repos-1.so.0[0x9c6f89]
/usr/local/lib/libsvn_repos-1.so.0(svn_repos_finish_report+0x49d)[0x9c745d]
/usr/lib/httpd/modules/mod_dav_svn.so(dav_svn__update_report+0xe9d)[0x538cdd]
/usr/lib/httpd/modules/mod_dav_svn.so[0x53be70]
/etc/httpd/modules/mod_dav.so[0x436cb1]
/usr/sbin/httpd(ap_run_handler+0x6d)[0x4ce85d]
/usr/sbin/httpd(ap_invoke_handler+0x68)[0x4d2208]
/usr/sbin/httpd(ap_process_request+0x1ae)[0x4de1fe]
/usr/sbin/httpd[0x4dafef]
/usr/sbin/httpd(ap_run_process_connection+0x6d)[0x4d671d]
/usr/sbin/httpd(ap_process_connection+0x4c)[0x4d681c]
/usr/sbin/httpd[0x4e27b2]
/usr/sbin/httpd[0x4e2ac1]
/usr/sbin/httpd[0x4e2b9a]
/usr/sbin/httpd(ap_mpm_run+0xb4b)[0x4e36fb]
/usr/sbin/httpd(main+0x8b7)[0x4ba0d7]
/lib/libc.so.6(__libc_start_main+0xdc)[0x6a2dec]
/usr/sbin/httpd[0x4b91d1]
======= Memory map: ========
[Wed Sep 12 10:55:57 2007] [notice] child pid 18823 exit signal Aborted (6)
---cut---

Help! Anyone have any suggestions for next steps? Have I missed an update somewhere that would make my life all better? Inquiring minds would love to know!

Could this somehow be related to the growing memory requirement? It always tanks at 27m resident RAM for the httpd process. I couldn't determine a good way to limit the actual memory use by the apache process to see if capping it lower would just make it crash sooner or if it would prevent the error...although I'm investigating how to set a ulimit on the spawned httpd process. I managed to get it to dump a core, but gdb just complains that the core isn't a properly formatted executable (which is odd). As you can tell, I'm not a coder, just a puzzled Op.

Thanks very much in advance.

-JB

[jbz]

So someone smarter than I did more digging after I collapsed from exhaustion. It turns out that there was a memory leak; it was coming from the DAV module. Adding the following to the DAV config lines in the vhost config seems to have fixed it:

SVNPathAuthz off

...which, AFAICT, tells the system not to reauthenticate for every subpath of the repo during large tree checkins, and hence mitigates the memory leak. I hope this helps somebody avoid the annoyance I went through.

[cpdohert]

I am having the exact same problem with almost the same config, but we're using CentOS 5 here and all packages have been installed from the CentOS 5 repositories, which gives us 1.4.2 for everything subversion-related.

I applied the

SVNPathAuthz off

directive and that solved the problem you've listed here, but now we are getting the error

PROPFIND request failed on '<repository path>'
PROPFIND of '<repository path>': Could not read status line: secure connection truncated (https://svn.myco.int)

which throws

[Fri Nov 23 13:03:43 2007] [notice] child pid 27786 exit signal Segmentation fault (11)

Into the apache error_log.

It does take a much larger project to generate this error, though. Can you tell us how you were able to determine the source of the leak?