Announcement

Collapse
No announcement yet.

subversion checkout on distrubuted filesytem

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • subversion checkout on distrubuted filesytem

    Hi Folks,

    We are using svn 1.8.11 and trying checkout on a distrubuted filesystem(DFS) like lustre.
    However it bails out. Mounting lustre filesystem with flock works to avoid filelocking with .sqllitedb but performance is very bad.
    Is there any solution for parallel/faster checkout on DFS?

    svn: E200030: sqlite[S10]: disk I/O error, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA foreign_keys=OFF;PRAGMA locking_mode = NORMAL;'
    svn: E155007: '/b/noormohamed/svn_test/DEV_COMMON_BRANCH/src' is not a working copy
    svn: E200030: sqlite[S10]: disk I/O error, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA foreign_keys=OFF;PRAGMA locking_mode = NORMAL;'
    svn: E200030: sqlite[S10]: disk I/O error, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA foreign_keys=OFF;PRAGMA locking_mode = NORMAL;'
    svn: E155007: '/b/noormohamed/svn_test/DEV_COMMON_BRANCH/src' is not a working copy
    svn: E200030: sqlite[S10]: disk I/O error, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA foreign_keys=OFF;PRAGMA locking_mode = NORMAL;'
    Last edited by DougR; 04-09-2018, 12:55 PM.

  • #2
    Looking here: http://grokbase.com/t/subversion/use...gh-the-network
    The discussion included at least 2 SVN committers (Philip Martin and Daniel Shahaf) - pay attention to their comments.
    It would appear in that case that byte-level locking was not properly supported.

    I'd run the same type of experiment that Philip Martin requested: 'strace svn co ...' and see what is causing the failure.

    If it's just the performance of the checkout that's the concern (when mounting DFS with flock support), then perhaps
    doing a checkout to a local FS and copying the result into the DFS? The subsequent "svn st -u" will still likely be a
    poor performer...

    Comment


    • #3
      Thanks Doug for the pointer. That's was our alternate plan to use local FS and copy to DFS.

      Comment


      • #4
        It that possbile to checkout only the meta data[sqllite db] in a local FS and svn export of sources in DFS and later create symlink to handle as a workspace?

        Comment


        • #5
          The sqllite db is not the only "private" part of the working copy. There is also the "pristine store" and some other files. They're all in the ".svn" directory at the working copy root.

          I just did a quick test by moving the .svn directory to a different location and putting a symlink in its place: did NOT work. The alternative would be to have the .svn directory there but symlinks to everything else - and that just can't work properly since symlinks are legal constructs in that directory.

          2 other possibilities:
          A. Try copying the ".svn" directory to a local FS and then mount that local FS over the ".svn" directory in the DFS (if mounting is allowable).
          B. Ask the SVN OpenSource community for help.

          Comment


          • #6
            I asked the advice of the OpenSource community and, if your problem is poor performance of SQLLite locking on DFS then you might want to look into SQLLite locking configuration changes. Per Philip Martin:

            If SQLite locking works at all then the client-side config option

            --config-option config:working-copy:exclusive-locking=true

            may help. It makes SQLite take less granular locks which improves
            performance, particularly on network filesystems, by reducing the SQLite
            overhead. The performance gain can be substantial.

            https://sqlite.org/pragma.html#pragma_locking_mode

            Exclusive locking also reduces the concurrency of Subversion operations:
            generally only one Subversion operation can run on a working copy at any
            one time, but lots of users never notice this.

            Another committer, Branko Čibej, noted that you can actually move the SQLLite database out of the ".svn" folder and onto an "ext4" file system and leave a symlink behind. Something like:

            mv .svn/wc.db /DFS/mywcdb/.
            ln -s /DFS/mywcdb/wc.db .svn/wc.db

            Finally, per another committer, Julian Foad, it's interesting to note that you can copy the ".svn" directory tree to some other location and then re-populate that location as a new working copy. I mention this because you might be able to "pre-populate" a working copy on your DFS by tar'ing the ".svn" directory up from an ext4 location, and then do an "svn update" to have it re-constitute itself. This might be helped by the suggestion by Philip Martin above.

            Comment

            Working...
            X