Newbie request for pointing in the right direction

[viptampa]

In short, I've been a minor SVN client user (via Tortoisesvn) for a few years, and this year I've setup my own VPS to help my game dev team manage code, compiled assets, and raw assets better.

I did setup my own SVN before via Dreamhost's webpanel, but now that I'm on my own VPS, it looks like I'll have to go command line.


As a rookie SVN admin, can someone refer me to some easy tutorials on how to setup repos, users, access controls for those users, and if there is a way to control the number of revisions kept (due to available space on the VPS).

Thanks a million
Sean

[mbooth]

The canonical users' and administrators' manual is the free book: http://svnbook.red-bean.com/en/1.7/index.html It also contains the command line reference. I'm sure you've Googled for tutorials

There is no way to control how many revisions the repository holds (there's no way to know how much space any given revision will consume until you commit it anyway.) Subversion has no business controlling disk quota anyway, you should use the operating systems' own tools for that. But you shouldn't keep compiled assets in source control anyway -- if that is what is using up your space -- they should be re-generated every time you do a build.

[viptampa]

Hmm, I find that odd that SVN has no way to trim off revisions that are xx amount revisions old or xx amount of days/months old. (even if I need to set a cron to do this manually)
When I said compiled, I should have said assembled.

To provide some further background, assembled assets are finalized & approved raw assets put in UPK containers that are needed for the level production process. Those UPKs while small individually, can get quite large once we start adding character models, textures, etc. Imagine storing 50 300mb UPKs , and then a 5 revisions of those. In theory, that could eat through 75GB if each had 5 or more revisions. Not everything needs to be stored in 1000'sz of revisions. However, keeping everyone in sync is important, hence the need of SVN

[mbooth]

Hmm, I find that odd that SVN has no way to trim off revisions that are xx amount revisions old or xx amount of days/months old. (even if I need to set a cron to do this manually)

To be efficient as possible with disk space used by the repository, each revision of a file is stored as a delta against an older revision of the file so it makes no sense to trim the oldest revisions -- they are needed to reconstruct the current revision of a given file. If a file doesn't change, no extra data is stored for that revision.

When I said compiled, I should have said assembled.

To provide some further background, assembled assets are finalized & approved raw assets put in UPK containers that are needed for the level production process. Those UPKs while small individually, can get quite large once we start adding character models, textures, etc. Imagine storing 50 300mb UPKs , and then a 5 revisions of those. In theory, that could eat through 75GB if each had 5 or more revisions. Not everything needs to be stored in 1000'sz of revisions. However, keeping everyone in sync is important, hence the need of SVN

Still, why store the UPKs in source control when you can generate them at build time? (By build time I mean do it as the first step of your level production process.) That's what continuous integration is for, after all.

[andyl]

Hmm, I find that odd that SVN has no way to trim off revisions that are xx amount revisions old or xx amount of days/months old. (even if I need to set a cron to do this manually)

Why? The purpose of a version control system is to preserve all your history. Intentionally deleting history just seems wrong in this context.

Additionally, every Subversion revision builds upon the revision(s) before. This trim can increase the size of your repository database, depending upon how things get moved around in the repository.

To provide some further background, assembled assets are finalized & approved raw assets put in UPK containers that are needed for the level production process. Those UPKs while small individually, can get quite large once we start adding character models, textures, etc. Imagine storing 50 300mb UPKs , and then a 5 revisions of those. In theory, that could eat through 75GB if each had 5 or more revisions. Not everything needs to be stored in 1000'sz of revisions. However, keeping everyone in sync is important, hence the need of SVN

Store your source in Subversion; for distribution of compiled/assembled artifacts & keeping them in sync across other systems, there are better & more efficient ways to do that. rsync is probably the most primitive, but the most readily-available.

[rivimey]

Mat,

The game dev process is quite different to the software dev process. Often the process of generating intermediate results is far from automatic, and so it is desireable to store these results: the "originals" are kept in case something needs reworking, but the intermediates are used while that is not the case. I understand completely why Sean would want to SVN these files.

Sean,

While the worst case scenario is as bad as you suggest, it is very much worst case and seriously unlikely. I have an SVN rep with roughly 500 revisions of a game environment + assets, and while it is 3.5GB in the working copy, the SVN repo for all 500 versions is only 2.6GB. The magic of deltas!

If you are serious about wanting to restrict disk usage, then impose quotas using the OS tools, based on the SVN repo, but I would advise against it: Find out first whether you have a problem and, at that stage, take action. In the meantime, remind your users not to store everything and anythng but only those files that are needed. For example, there's really no point in storing local backup versions of files : if you have myfile.bin in SVN, don't also store myfile.bak as well. Or, if you have a tool that takes file.mod as input and creates file.out and file.log, don't store the .log file.

HTH
Ruth

[viptampa]

Ruth,

Thank you for the clarification. I was unaware SVN used partity/deltas to construct all the revisions. I was under the impression of each revision of a binary file was copied in whole thus eating storage space.

Thanks for the direction pointing. Yea, we weren't planning on storing backups in SVN, just current work so that everyone can be on the same page.

[viptampa]

Still, why store the UPKs in source control when you can generate them at build time? (By build time I mean do it as the first step of your level production process.) That's what continuous integration is for, after all.

UPKs cannot generated at build time. UPKs are custom asset containers that are called upon by the game engine during editing and during build time. They have to be generated MANUALLY though before hand.

[andyl]

UPKs cannot generated at build time. UPKs are custom asset containers that are called upon by the game engine during editing and during build time. They have to be generated MANUALLY though before hand.

Mat's point stands. If the UPKs can be generated from other (less space-hungry) resources which are stored in the repository, storing the UPKs in the repository is redundant and wasteful (space, bandwidth). Generate new UPKs for items which have been changed after each checkout/update (you'll need to script around svn update or use the client hooks available in TortoiseSVN).

[viptampa]

Sigh, you don't understand how UPKs work and it sounds I may be talking to the wrong crowd.

Again, UPKs are not generated... they are assembled manually from RAW art/model/audio assets. Those raw art/model/audio assets need to be kept track of as if a developer leaves, we need to be able to pick up and go from where he/she left off. Generally raw game content are larger than UPKs (UPKs use compression). We need to keep revisions of UPKs as if we make a change to a material or property for a model or audio sound cue within that UPK and it fails to work as expected, we need to be able to roll back a revision. I've talked to game project leads on other projects, and they store their projects just usually on a colocation/dedicated server. Since I don't have that, I opted for a centralized VPS. From what Ruth is saying, it looks like the deltas process of SVN will save me the hassle of attempting to maintain repos as I originally that I would need.

I do appreciate everyone's time even if we might not understand one another.

[rivimey]

Sean,

I'd suggest having a go on a limited trial of some sort to see how things work for you. I think svn can answer your need and there are lots of good tools around it.

Ruth

[mbooth]

Sigh, you don't understand how UPKs work and it sounds I may be talking to the wrong crowd.

Again, UPKs are not generated... they are assembled manually from RAW art/model/audio assets. Those raw art/model/audio assets need to be kept track of as if a developer leaves, we need to be able to pick up and go from where he/she left off.

Game development is not a special case. I still don't see why you have to do it manually. There is a tool to assemble a UPK and your developer is maintaining a list of files/revisions that goes into them, right? Why not store that list in SVN and script the tool to assemble the UPK from the list?