Using the pre-commit hook to restrict the size of the submitted file failed

[yipianlvye]

Dear Sir:
How do you do?

1.Environment description:
Visual SVN Server 2.7.13, installed on Windows Server 2007 SP2 operating system.
Now I create a version library named test. It works fine.

2.My needs:
I want to limit the size of the submitted files.

3.My operation:
1) I copy the file named pre-commit.tmpl and save it as file named pre-commit.sh in hooks directory.
2) Modify the file named pre-commit.sh as follows:
#!/bin/bash

REPOS="$1"
TXN="$2"

SVNLOOK="C:/Program Files/VisualSVN Server/bin/svnlook"
#SVNLOOK=/usr/bin/svnlook
MAX_SIZE=512000

files=$($SVNLOOK changed -t $TXN $REPOS | awk '{print $2}')

# check check
#if [[ $files =~ "project_nuli" ]];then
for f in $files
do
# check file size
filesize=$($SVNLOOK cat -t $TXN $REPOS $f | wc -c)
if [ $filesize -gt $MAX_SIZE ] ; then
echo "File $f is too large (must <= $MAX_SIZE)" >> /dev/stderr
exit 1
fi
done
#fi
exit 0
3) Restart the SVN service

4.actual result:
Then I submit a 46k file, but it succeeded. The new pre-commit hook seems not to work?

What’s wrong?
Fervently hope expert’s advice. Your early reply will be highly appreciated.

Thank you so much!
Yours
yipianlvye
June 30, 2018

[DougR]

On Linux the name of the file must be exactly "pre-commit" (not "pre-commit.sh"). I'm not an expert on Windows, but the extension that you use must be registered with Windows as an executable. If you've got ".sh" so registered then you get to move on to the next part of debugging: checking to see what happened. I normally do this by turning on the "shell echo" capability (e.g. "set -x") and then making sure that the output is redirected to some temporary file that I can evaluate later. In general, it might look like:

set -x
for x in 1; do
put your code here
done > /path/to/some/temp/file 2>&1

Hopefully you'll then figure out what's going wrong.

[yipianlvye]

Thank you for your reply.
I’ve solved this problem with the help of RD engineers. They found that the script could not run, mainly because the two commands (awk and wc) under Linux did not have corresponding commands under Windows. They rewritten the script as a Windows script and the problem was solved.

[yipianlvye]

Now I meet another question.
Another repository named SW.
URL: https://10.7.0.84:8443/svn/SW/软件事业部/01项目区/01工程项目/2017/甘肃掌上操作票专家系统项目/03.Code/3.2.Trunk/P70
Checkout failed, Error: missing update-report close tag

I execute ‘svnadmin verify E:/Repos/SW’ command on the server side, and the result is as follows:
Verified revision 63848.
svnadmin:E140001:zlib <uncompress>:corrupt dataecompression of svndiff data failed
It seems that the 63849 version data has been corrupted.
Version 63849 is as follows:
[IMG]file:///C:%5CUsers%5Cyyc%5CAppData%5CLocal%5CTemp%5Cmsohtm lclip1%5C01%5Cclip_image002.jpg[/IMG]
If the version's data is corrupted, what can we do to fix it or remove it completely? The svnadmin dump operation also failed.

[DougR]

Use the dump and specify a range from 0-(N-1) and (N+1)-last. Then load up a new repo. This may not work if N+1 refers to N so you may need to iterate forward.

Any working copy referring to any is the renumberred revisions will need to be discarded and re checked out (save the work outside the WC.

Lot of pain. Repo corruption is very rare. You should look into why. Check windows error logs, etc. Prevent the issue for happening again.

[yipianlvye]

According to your method, the following commands are successful, and I get 3 dump files:
svnadmin dump E:\Repos\SW -r 0:63848 > G:\SW1.dump
svnadmin dump E:\Repos\SW -r 63850:66598 --incremental > G:\SW2.dump
svnadmin dump E:\Repos\SW -r 66600:88013 --incremental > G:\SW3.dump
But the execution of the following command meets an error:
svnadmin load E:\Repos\SW1 < G:\SW1.dump

Error information is below:
---Committed revision 2327 >>>

<<< Started new transaction, based on original revision 2328
* editing path:软件事业部/01项目区/05测试库/Project1 ...svnadmin:E200020
:Invalid svn:mergeinfo value
svnadmin:E200020:Unable to parse reversed revision range '96798-2321'
svnadmin:E200020:Unable to parse reversed revision range '96798-2321'

What’s wrong?
Hope for your suggest. Thank you so much!

[DougR]

That’s messed up meta-data. I’ve seen an old SVN bug that could have caused the issue, but I have no idea about Visual SVN. You should contact the Visual SVN folks for help.

[davidvazqueztds]

The script does not get the filenames which contain blank spaces. Anyone know how to read filenames with spaces? I tryed including " ", ' ', and \ before each space.
For example:
It get the size of "file.pdf", but nor for "file 1.pdf". I tryed including " " to the name, like "file .pdf", 'file .pdf', "file\ .pdf"

Maybe SVNLOOK does not get filenames with spaces and thereīs nothing to do...

[DougR]

Handling white-space within Linux scripting takes a lot of care in proper quoting. You have not shared the script so I cannot say where things went wrong.

[davidvazqueztds]

Thatīs right, sorry! I forgot it

This is my script:
#!bin/bash
REPOS="$1"
TXN="$2"

SVNLOOK=/usr/bin/svnlook
MAX_SIZE=262144000
#files=$($SVNLOOK changed -t $TXN $REPOS | cut -d A -f2 | cut -d U -f2 | cut -c4-
#files=$($SVNLOOK changed -t $TXN $REPOS | cut -d A -f2 | cut -d U -f2 | cut -c4- | sed "s/^/"/;s/$/"/")
#files=$($SVNLOOK changed -t $TXN $REPOS | cut -d A -f2 | cut -d U -f2 | cut -c4- | sed 's/ /\\\$/g' | tr $ " ")
files=$($SVNLOOK changed -t $TXN $REPOS | awk '{print $2}')
#echo "ARCHIVOS: $REPOS $TXN $files" >> /home/user/prueba.log
for f in $files
do
filesize=$($SVNLOOK cat -t $TXN $REPOS $f | wc -c)
#echo "PRUEBA CON FILESIZE: $filesize" >> /home/user/prueba.log
if [ $filesize -gt $MAX_SIZE ]; then
echo "File $f is too large, must be lower than 250MB">&2
exit 1
fi
done
exit 0


I tryed different ways of get results about SVNLOOK changed (commented options). As you see, I insert " " (and also can insert ' ' just changing it on my script). I also tryed with inserting \ before each space. Im checking the results of the script into log files, and Iīm introducing " " and ' ' properly, but SVNLOOK cat does not get it...

Anyway, using the loop will be a problem if it check spaces, because it wonīt get the proper file, it will get each word after each space as a different word.

[DougR]

First, you could always use the "--xml" option to "svnlook" and then use an xml parser. You'd likely need to change languages away from bash.

Second, using "cut" is, well, tricky: I try to avoid it.

Third, because you're using "awk '{print $2}'" the data you're looking for is already broken since awk, by default, uses white-space to separate fields.

#!/bin/bash --noprofile
REPOS="$1"
TXN="$2"

SVNLOOK=/usr/bin/svnlook
MAX_SIZE=262144000
$SVNLOOK changed -t "$TXN" "$REPOS" | sed -e '/^D/d' -e 's/^....//' | \
while read afile; do
case "$afile" in
*/) continue;; # skip directories
esac
filesize=$($SVNLOOK cat -t "$TXN" "$REPOS" "$afile" | wc -c)
#echo "PRUEBA CON FILESIZE: $filesize" >> /home/user/prueba.log
if [[ $filesize -gt $MAX_SIZE ]]; then
echo "File $afile is too large, must be lower than 250MB">&2
exit 1
fi
done
exit 0

Give that a try (might be some syntax mistakes - I don't have time right now to test it.

[davidvazqueztds]

I tryed it:

#!bin/bash
REPOS="$1"
TXN="$2"

SVNLOOK=/usr/bin/svnlook
MAX_SIZE=104857600 #250MB

files=$($SVNLOOK changed -t "$TXN" "$REPOS" | sed -e '/^D/d' -e 's/^....//')
#files=$($SVNLOOK changed -t $TXN $REPOS | cut -d A -f2 | cut -d U -f2 | cut -c4- | sed "s/^/"/;s/$/"/")
#files=$($SVNLOOK changed -t $TXN $REPOS | awk {'print $2'})
#files=$($SVNLOOK changed -t $TXN $REPOS | cut -d A -f2 | cut -d U -f2 | cut -c4- | sed "s/^/\'/;s/$/\'/" | sed 's/ /\\\$/g' | tr $ " ")
echo "ARCHIVOS:$files" >> /home/user/prueba.log

#for f in $files
#do
while read files
do
case $files in */)
echo "CASE:$files" >> /home/user/prueba.log
continue
;;
esac
filesize=$($SVNLOOK cat -t $TXN $REPOS $files | wc -c)
#filesize=$($SVNLOOK filesize -t $TXN $REPOS $files)
#filesize=$(du -bs $files | cut -f 1)
echo "PRUEBA CON FILESIZE: $filesize" >> /home/user/prueba.log
if [ $filesize -gt $MAX_SIZE ]; then
echo "SE CUMPLE" >> /home/user/prueba.log
echo "File $f is too large, must be lower than 250MB">&2
exit 1
fi
done
#done
exit 0

It does not work. The script does not reach the filesize function. In that case, is not getting the size of the file.
I donīt know why actually, I tryed it with filename without spaces and with spaces. No way to reach filesize... I think problem is in "while read", because if I use "for" instead, it works, but it does not go trhough case statment
any idea?

I think even if script works, the problem will be that SVNLOOK does not get file size if name contain spaces.

[DougR]

You can't use an inline function like $() when you need to parse file names that have white-space.
So the "files=$()" is just plain not going to work.
In the code you posed up there's no $ in front of the "files" in "while read files". And you're not feeding anything into it so no, it's not going to work.

Did you try using the script as I posted it above? What happened?

[davidvazqueztds]

Yes, I tryed your script but it does not work...
because "afile" variable is empty. In your code, you did not create the variable or save information inside, so code is not working.
Im doing it like you now:

#!bin/bash
REPOS="$1"
TXN="$2"

SVNLOOK=/usr/bin/svnlook
MAX_SIZE=104857600 #250MB
#files=$($SVNLOOK changed -t "$TXN" "$REPOS" | cut -d A -f2 | cut -d U -f2 | cut -c4-)
$SVNLOOK changed -t "$TXN" "$REPOS" | sed -e '/^D/d' -e 's/^....//'
#files=$($SVNLOOK changed -t $TXN $REPOS | cut -d A -f2 | cut -d U -f2 | cut -c4- | sed "s/^/"/;s/$/"/")
#files=$($SVNLOOK changed -t $TXN $REPOS | awk {'print $2'})
#files=$($SVNLOOK changed -t $TXN $REPOS | cut -d A -f2 | cut -d U -f2 | cut -c4- | sed "s/^/\'/;s/$/\'/" | sed 's/ /\\\$/g' | tr $ " ")
echo "ARCHIVOS:$files" >> /home/user/prueba.log

#for f in $files
#do
while read afile; do
echo "WHILE:$afile" >> /home/user/prueba.log
case $afile in */)
echo "CASE:$afile" >> /home/user/prueba.log
continue
;;
esac
filesize=$($SVNLOOK cat -t $TXN $REPOS $afile | wc -c)
#filesize=$($SVNLOOK filesize -t $TXN $REPOS $files)
#filesize=$(du -bs $files | cut -f 1)
echo "PRUEBA CON FILESIZE: $filesize" >> /home/user/prueba.log
if [ $filesize -gt $MAX_SIZE ]; then
echo "SE CUMPLE" >> /home/user/prueba.log
echo "File $f is too large, must be lower than 250MB">&2
exit 1
fi
#done < "$files"
done
exit 0

[DougR]

In my script the line just before "while read afile; do" is:

$SVNLOOK changed -t "$TXN" "$REPOS" | sed -e '/^D/d' -e 's/^....//' | \

That line ends in a pipe followed by a backslash which should then be followed by a newline (EOL).

Or, you can combine those 2 lines:

$SVNLOOK changed -t "$TXN" "$REPOS" | sed -e '/^D/d' -e 's/^....//' | while read afile; do

NOTE: the quoting for the sed commands should be apostrophe (single quotes) - not something silly that a copy/paste might turn them into. Same with the double-quotes.