Announcement

Collapse
No announcement yet.

SVN + HTTPD - 500 Internal Server Error

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • SVN + HTTPD - 500 Internal Server Error

    Backstory:
    We decided to migrate the SVN from On-Prem to Cloud.
    Both servers are CentOS 7 and the SVN version On-Prem is 1.8.15 while on Cloud it's 1.8.19;
    The access protocol changed from SVN (port 3690) to HTTPS (443), so the httpd setup is a novelty.

    For the migration of the repository, I've tried doing a plain old 'rsync' between the servers to move the whole repository, and it worked since the functionality & all the revisions were there, however I still got the same error.
    I thought it may be some kind of DB issue, so I then used the SVN-native 'svnadmin dump' and 'svnadmin load' commands to import the repository. The issue still persists.

    I am using SVN accessed using HTTPS through Apache HTTPD. Everything seems to work fine and all the functionality is there, but after several checkouts I start getting a 500 Internal Server Error.

    Currently, the issue is caused by a Jenkins pipeline which checkouts from SVN, here is the outputted error:
    Code:
    ERROR: Failed to check out https://svn-repo/path/to/files org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS of '/path/to/files': 500 Internal Server Error (https://svn-repo) svn: E175002: OPTIONS request failed on '/path/to/files'
    The reason why I don't think it's a problem from the client (Jenkins) side at the moment is because the same error happened to me when checking out from my PC SVN client.

    Here are the logs from HTTPD:
    Code:
    10.10.10.16 - - [17/Aug/2020:12:45:21 +0300] "OPTIONS /path/to/files HTTP/1. 1" 401 381
    10.10.10.16 - user [17/Aug/2020:12:45:21 +0300] "OPTIONS /path/to/files HTTP/1.1" 500 541
    As you can see, I receive a 401 before getting the 500, but as I said the checkouts occur one after the other so it couldn't have checked out something previously if the authorization was invalid (the permissions for the whole repo are identical, not path-based).

  • #2
    You'll always get a 401 first because of the definition of the WEBDAV protocol: it allows unauthenticated so will always try unauthenticated 1st. If it gets back a 401 then it will send your credentials.

    So the problem is that the "OPTIONS" request is failing. Have you checked the Apache "error_log" file (or, more likely, "ssl_error_log" file)? If so, what's in it?

    This is likely just an Apache setup issue, but you should run "svnadmin verify --keep-going" on the repository to verify that it is healthy.

    That said, SVN 1.8 is ancient and completely out of support. In fact, now that SVN 1.14 has shipped, even SVN 1.9 is out of support by the community.
    You should consider SVN 1.14.
    Last edited by DougR; 09-03-2020, 07:09 PM.

    Comment


    • #3
      Originally posted by DougR View Post
      You'll always get a 401 first because of the definition of the WEBSAV protocol: it allows unauthenticated so will always try unauthenticated 1st. If it gets back a 401 then it will send your credentials.

      So the problem is that the "OPTIONS" request is failing. Have you checked the Apache "error_log" file (or, more likely, "ssl_error_log" file)? If so, what's in it?

      This is likely just an Apache setup issue, but you should run "svnadmin verify --keep-going" on the repository to verify that it is healthy.

      That said, SVN 1.8 is ancient and completely out of support. In fact, now that SVN 1.14 has shipped, even SVN 1.9 is out of support by the community.
      You should consider SVN 1.14.
      Thank you for the quick response.
      I ran "svnadmin verify" and everything seems to be fine.
      Regarding the error log file, it is mostly empty with some errors that aren't really relevant to the checkouts and don't correspond in terms of the time.
      After doing so many checks I really think it's some kind of networking issue because it usually happens when checking out a lot of data, maybe the networking spike is too much for the AWS EC2 instance.

      Regarding upgrading to 1.14, is there something that needs to be changed from the repo perspective?

      Comment


      • #4
        Got to ask, how old is the repo? Perhaps a redacted output of "svnadmin info /repo/path" ? Or just the output from "cat /repo/path/format /repo/path/db/format"?

        In general, SVN 1.14 on the server should be able to serve all repositories without requiring anything be changed. That said, there are 2 paths to gain additional features: "svnadmin upgrade" or a "dump/load". The "dump/load" path gains all possible new behaviors whereas the upgrade only gets some of them. See the release notes for SVN 1.9 (where the majority of repository format changes last took place) for more details.

        I would not expect a 500 error due to AWS EC2 networking limitations...

        Comment


        • #5
          Originally posted by DougR View Post
          Got to ask, how old is the repo? Perhaps a redacted output of "svnadmin info /repo/path" ? Or just the output from "cat /repo/path/format /repo/path/db/format"?

          In general, SVN 1.14 on the server should be able to serve all repositories without requiring anything be changed. That said, there are 2 paths to gain additional features: "svnadmin upgrade" or a "dump/load". The "dump/load" path gains all possible new behaviors whereas the upgrade only gets some of them. See the release notes for SVN 1.9 (where the majority of repository format changes last took place) for more details.

          I would not expect a 500 error due to AWS EC2 networking limitations...
          The output of "cat /repo/path/format /repo/path/db/format":
          Code:
          6
          layout sharded 1000
          What do you can cause random 500 internal server errors? It doesn't happen on a specific path, and on rare occasions the pipeline does manage to checkout everything but much slower than on the On-Prem.

          Comment


          • #6
            That's an "native SVN 1.8 format" so should be fine with SVN 1.14 serving it out.

            In terms of "much slower", that's expected because WEBDAV is rather chatty and latency will have a huge impact on your measured command times. As an example "svn blame" is known to be very expensive.

            In terms of 500 "internal server" errors, check that every artifact (directory, file, etc.) within the repository is owned properly and has proper permissions.

            Also, you should turn off "SELinux" if its enabled - unless you want to go through the trouble of tuning it for use (e.g. putting it in permissive mode, running through a complete set of operations, then turning the captured data into a policy and setting the policy in place).

            Comment


            • #7
              Originally posted by DougR View Post
              That's an "native SVN 1.8 format" so should be fine with SVN 1.14 serving it out.

              In terms of "much slower", that's expected because WEBDAV is rather chatty and latency will have a huge impact on your measured command times. As an example "svn blame" is known to be very expensive.

              In terms of 500 "internal server" errors, check that every artifact (directory, file, etc.) within the repository is owned properly and has proper permissions.

              Also, you should turn off "SELinux" if its enabled - unless you want to go through the trouble of tuning it for use (e.g. putting it in permissive mode, running through a complete set of operations, then turning the captured data into a policy and setting the policy in place).
              Hi Doug, thanks once again for your response.
              I understand that from your perspective this is just an issue you are seeing now, but I've struggled with it for several weeks and have done all the checks / changes you can imagine, permissions & SELinux being one of the first. The issue here is not permissions nor is it an issue with the DB because I can checkout from the exact same places when I do it in small chunks.

              After the checkouts I visited the exact same page there was a problem checking out from and got a 500 from my browser. After several seconds I refreshed the page and it was fine.

              Comment


              • #8
                Check your /var/log/messages file to see if httpd is getting killed by the kernel memory daemon. If so then you're running out of memory and will need to decrease the HTTP config tunables or add more RAM or add more hard swap (or some combo of them).

                Comment


                • #9
                  Originally posted by DougR View Post
                  Check your /var/log/messages file to see if httpd is getting killed by the kernel memory daemon. If so then you're running out of memory and will need to decrease the HTTP config tunables or add more RAM or add more hard swap (or some combo of them).
                  Nothing related in there, unfortunately.

                  Comment


                  • #10
                    It's been brought to my attention that 'SVNAllowBulkUpdates On' could be the cause of this issue.
                    I tried running the pipeline both with 'Prefer' and 'Off', however that did not fix the issue.

                    Comment


                    • #11
                      While you're there you might try turning off the V2 protocol: "SVNAdvertiseV2Protocol off"
                      (see here for more things to try: http://svnbook.red-bean.com/en/1.8/s...ef.mod_dav_svn )

                      Also, make sure NOT to load the Apache compression (commend out: #LoadModule deflate_module modules/mod_deflate.so).

                      Comment


                      • #12
                        Originally posted by DougR View Post
                        While you're there you might try turning off the V2 protocol: "SVNAdvertiseV2Protocol off"
                        (see here for more things to try: http://svnbook.red-bean.com/en/1.8/s...ef.mod_dav_svn )

                        Also, make sure NOT to load the Apache compression (commend out: #LoadModule deflate_module modules/mod_deflate.so).
                        Added the following to the config:
                        Code:
                        # Config for trying to fix 500 errors
                        SVNAllowBulkUpdates Off
                        SVNAdvertiseV2Protocol Off
                        SVNCacheFullTexts On
                        SVNCacheTextDeltas On
                        Also, commented out the module you stated. Still does not work.
                        Once again, I went to the page from which I received a 500 internal server error whilst checking out using 'GET'. It loaded for several seconds (maybe a bit more than several), and after a refresh (F5) it swiftly loaded the page properly.

                        Got these in my error log:
                        Code:
                        [Thu Sep 03 20:21:56.985800 2020] [dav:error] [pid 19551] [client 10.11.11.11:49438] Provider encountered an error while streaming a REPORT response. [500, #0]
                        [Thu Sep 03 20:21:56.985826 2020] [dav:error] [pid 19551] [client 10.11.11.11:49438] A failure occurred while driving the update report editor [500, #104]

                        Comment


                        • #13
                          There are a series of things to double-check here: https://stackoverflow.com/questions/...s-sporadically
                          Yes, you've already done some of them. I was wondering if the "AuthDigestNonceLifetime -1" might help? There are some others down lower - check them out too.

                          Comment


                          • #14
                            Originally posted by DougR View Post
                            There are a series of things to double-check here: https://stackoverflow.com/questions/...s-sporadically
                            Yes, you've already done some of them. I was wondering if the "AuthDigestNonceLifetime -1" might help? There are some others down lower - check them out too.
                            Tried all the directives that were recommended in the threat.
                            The only one I didn't do was the "SVNCompressionLevel" because it rejected it for some reason with the following error:
                            Code:
                            SVNCompressionLevel not allowed here
                            Even though I put it in the same place as the other configs.

                            Comment


                            • #15
                              Per here: http://svnbook.red-bean.com/en/1.8/s...fig.httpd.perf

                              SVN 1.8.anything should have the "SVNCompressionLevel" tunable - placed along with the "SVNCacheTextDeltas", et. al. If your SVN is complaining about it then that would indicate that there's something wrong about how it was compliled up... Upgrade it with something newer (and actually supported by the community - SVN 1.8 is way out of support). The 2 LTS (long term support) releases currently available are 1.10.x and 1.14.x.

                              Comment

                              Working...
                              X