mann: Ah, good one. Definitely a problem there. Now I'm starting to see why the people on #revctrl spend so much time talking about cases like this. :) hmm. I suppose "mv foo/ bar/" puts an attribute on bar saying "i used to be foo". then the second rename has to notice that attribute when putting the attribute on frotz.c saying "i used to be foo/baz.c". yeah, i've never thought seriously about it before, so i'm discovering the complexities by trial and error mann: I was just typing something like that. :-) mann: I think there's even more complexity in deciding what to do when trying to merge changes across renames, but I'm expecting that there are algorithms in the other free systems that can be apropriated for that. hmm. this is sort of a lazy way of implementing something that obviously works: upon checkout, mark everything in the wd with an attribute saying "my name used to be what/ev/er". mann: Hmm, that's another possible idea. It would have the side effect of creating mutable directories for everything though, which again concerns me for large packages, as the vadvance time is proportional to the number of mutable directories. i suppose for change merging purposes one might like it if each "thing" (conceived of as a mutable entity with a history) were identified by some uniqueID, and the properties of a version of a thing included both its contents and its name. then renaming is an edit like any other. sorry i'm popping out with ideas too fast. back to the "this is sort of a lazy way"... what i meant there is not that we would literally tag everything like that. rather, imagine that we did, but don't actually generate an object's old name until the moment you rename it. then anything sitting in the tree without an old name tag has not been renamed relative to its parent mann: Seems like it could work. when you do rename something, you have to search up through its parents and catenate their old name tags (or their current name if they don' t have an old name tag) to get the child's tag. hmm, something that's actually tagged has its complete old name, so you don't search up beyond that. i think this algorithm is correct, but it's off the top of my head. mann: It sounds right to me. The idea of a unique ID for files reminds me of something I heard about BK. Something about files that get deleted essentially being renamed to a "deleted" area but keeping their unique ID. It has something to do with merging changes from different branches and delete and undelted operations. Anyway, it's really late, and I need to get some sleep. nice brainstorming with you! mann: Yes, thanks! Since everyone else keeps such unsociable waking hours ;P, I have a list of questions and responses I'm afraid :) 1. "What if instead the evaluator preserved the mtime of files it ships?" If this were the *actual* mtime, i.e. not updated across checkout/checkin unless the file actually changed, then I guess that would Just Work ? BUT it would also have to change if, e.g. you changed the permissions of the file within /vesta/, no? I think, IMHO, it would be a much better default situation than we have at the moment, in that what we have now (the last-build-time rather than actual mtime) ...is rather anti-social w.r.t. the rest of the OS and apps - it defeats a lot of systems, from webservers to incremental backup apps, which rely on this ...IMHO and IME rather flakey attribute from the FS (but then I came from DOS first, so I'm used to mtime being a very variable concept ;)) 2. "it would have been nice if we'd thought of a way to design vesta so that the repository and builder were separate modules that you could mix and match with other things." Actually, if you had, I would probable never have adopted vesta, and certainly never adopted the build system ...and, in so doing, missed out on the big advantages. ...I was only looking for "a fast, robust, stable, accurate, OSS SCM". My expectations were high, having used big ugly (but powerful and stable) ...tools in corporate dev (e.g. IBM) - and absolutely detesting CVS having found the stupid mind-blowing bugs like the complete lack of cvs is evil ...synch locks allowing ANY simultaneous checkin to corrupt your source, which had more than once caused almost lethal damage to projects I was managing. So...although it is tempting to say "if it were less all-or-nothing, more people might use some of it" I firmly believe that we're on the better path... ...although it's harder to sell (conceptually speaking), it's got a much higher retention rate. ...and let's not forget that these days most people *really* want complete systems, not hodge-podges, because so rarely does a pick-n-mix OSS system ...actually work. So many incompatibilities, so many minor bugs that propagate and escalate as they go from app to app ...(e.g. OS X allows *unescaped* question marks and asterisks in file names. This breaks rather a lot of linux server apps :( ) So a "absolutely guaranteed to work very very well together" SCM + build system *with extra bonuses from the integration* is a very valuable tool indeed. 3. "stupid vcheckout trick" - could we have a wiki page on that? When to use it, and what it does :) Because I have no idea what the point is, but suspect it could be useful 4. Stupid question: what do you get from tracking renames? It's only something I've encountered as really useful in delta-compression dependent systems. which vesta...isn;t? 5. ...going further, how long will it be before the attribs system becomes overburdened with too much complex usage for it's simplicity - do we need to move to a richer, more designed, system? ...this is something that's been worrying me for some time. It's IME a problem waiting to happen, like putting a perl interpreter on every field in a ...data-collecting app and allowing anyone to type anything and evaluating it before recording it. ...very powerful, but like being locked in a dark room full of nooses and open trapdoors, as far as long-term and large-project maintainability goes. 6. I think we need a wiki page on "Attribute programming with vesta" because the flexibility ets you do a heck of a lot without needing to know anything ....about the core of vesta other than user-level knowledge of tools etc. If you don't mind doing mucho shell scripting. 7. And I think another, perhaps, "Things you should never do" ...starting with - mann: renaming one's working directory... 8. Could we have a vesta wiki page on sids, fingerprints, and hashes? What they each do, compare and contrast, how they work, where vesta depends on them, why, and how they're stored in the repos as funnily named dirs :) 9. What is COW? 10. Do we have a wiki page for "the new "add-ons" design" ? 11. I suspect we also need a wiki page to cover the "vcheckout -F" to ensure people understand the issues and don't go writing a HOWTO that states "in vesta, you always use -F. Just something you need to remember, or put into your own tiny shell script, called e.g. "co", that ...calls the vcheckout with this mandatory argument" "Client exiting" 12. "how compression method selection might work. Client sends a lits of integer compression method codes in order of preference" ...I recommend adopting the HTTP encoding-negotiation protocol, unless there's particular reason not to. input focus 1 mike 0 ...since it's already documented and it's actually very neatly readable in progress. Although IIRC it leaves the server room for choosing the algo eventually used ...to select one. 13. "git mailing list has had 1317 messages in less than 1 month. vesta-devel has had 1209 in it's entire history." yes, but GIT mailing list has Linus (*scream* *swoon*) on it :) ...that's merely because vesta devs aren't evangelists :P. ...without wanting to put anyone down, there's clearly no marketing professionals involved in vesta. ...I regularly run into the PerForce marketing army at conferences and shows, and a lot of mileage could be gained merely from copying a few of their ...techniques. ...To date, vesta has lacked a marketing strategy. One could be put together fairly easily, and most of it would not be hard nor expensive to follow-through ...on - but the real question is how much the core people involved in vesta already actually want to grow adoption rates etc? ...Personally I have 3 motivations: 1. Get more people contributing so I benefit :P. 2. Increase the number of people skilled with vesta so my hiring is easier ...3. Make it easier for me to convert organizations where I work to use vesta, where it's appropriate. ...and, since nobody expects the Spanish Inquisition, 4. save humanity from **** cvs and all the other crummy SCM's that waste people's lives and 5. Rant about NFS at every available opportunity G,D,RVVVF. How is 5. a motivation for getting more people using vesta? Bigger audience? was planning on marketing DVDs featuring teh ranting talents of various Vesta devs. Penfold: but your DVDs won't play in my DVD player. :( adammartin: wow... that was quite the list. I can answer one of them: http://wiki.vestasys.org/std_env see the bottom of the page. scottven: they will if I make 'em region 0 :) Yow, what a list to wake up to. :) adammartin: Yes, I meant the actualy mtime of the file. (Of the shortid, actually.) adammartin: For sources this would be the last time the user wrote the source befor vadvancing it. For deriveds, it would be whenever the tool finished writing it. adammartin: The only permissions change which would affect this is turning on or off the execute bit (because that creates a new shortid). adammartin: I think some wiki pages on attributes would be very handy, but I'm not sure what you mean by "attribute programming". adammartin: Do you mean "how to write scripts using vattrib"? One thing I would like to see is a list of all the attributes used by the core Vesta tools and what they do/mean and what set/uses them. adammartin: shortids and fingerprints are kind of getting into the realm of the internals of the system, but I don't see any reason why we can't put some of that stuff in the wiki. adammartin: COW = Copy On Write adammartin: When you perform a checkout, the repository doesn't copy any files. The storage for the files in the working directory is the same as the immutable files in the old version. adammartin: When you perform a write to a file in a working directory, then the repository makes a mutable copy of it. adammartin: What would the wiki page about "vcheckout -F" say other than what's in the man page? adammartin: I guess the man page really just documents the "how" of that feature, not the "why". :-/ adammartin: About the HTTP encoding-negotiation protocol: what's special/useful about it? Could you give a brief description of how it works? adammartin: I'd prefer to not turn this into something that takes multiple round-trips between the client (destination repository) and server (source repository). adammartin: The idea we've implemented now is all wrapped up in a single RPC. adammartin: Yes, you're absolutely right about evangelism. I try in technical circles, but my efforts have been small and I'm no marketing person. You can see the DEC legacy here: "A triumph of engineering over marleting". :-/ My main interests are: 1. Getting more contributors. It's a large code base. There's lots to do. And I won't view this "experiment" in free software to be a success until my group is doing 50% or less of the maintenance effort on Vesta. We have briefly discussed the idea of running a Google ad campaign, using the SourceForge donation system to fund it. But I've never gotten around to setting that up. I've actually been thinking for a while that we should try to sell more "serious" engineering disciplines on Vesta. People doing mechnical, industrial, architecutral, nuclear, and biological engineering. Places where mistakes are costly (similar to the microprocessor industry where I work). I have *no* idea how to go about doing that though. scottven: I think I now have a pkf2vesta that provides messages and is backwards compatible. xorian: tool intigration would be a big win. Need to convince AutoCAD or some such tool to use a vesta backen. or maybe ship with a mathmatica bridge. scottven: Heh, intersting idea. I've always though of Mathematica as an interactive tool, but maybe that's just because I used the early versions on the Mac. xorian: erm... I think I ment matlab Jumping back a bit (I tried to get them all in order, really I did): adammartin: On tracking renames, I'm trying to think ahead a alittle bit. Implementing automerge is high on my list of things to do. adammartin: However, I know that people who work on the sophisticated modern VCSes (Darcs, Codeville, Monotone, etc.) worry a lot about being able to handle cases like: User A edits file in a branch, User B renames file in main-line, now merge. It seems to me that we could get the repository to track renames for us (and I think Tim and I worked out a pretty good algorithm for doing so last night). This could actually give us an advantage over other systems where you have to explicitly tell the VCS "I renamed this file". adammartin: As to "the attribs system becomes overburdened with too much complex usage": what's the alternative? I think I understand what you're saying in that there's only so much you can do with the attribute name:set-of-values system, and we may suffer from the "to a man with a hammer everything looks like a nail" problem. But what alternative would you propose? The simplicity of the current system is also powerful in that it's easy to understand. . o O ( except for that timestamp issue ) Also, I suppose as we come up with new things to store in attributes there's more of them which should be documented somewhere and more things people might want to understand. scottven: That's high on my list for release/13, though lower than the two remaining fsync bugs we discovered on the flaky server. I mean the timestamp thing should be an afternoon fix. I just have to work out a couple minor details in the implementation. xorian: are we taling aobut the same timestamp thing? mann: http://better-scm.berlios.de/vesta/ has been updated, BTW. xorian: I'm talking about it for attribute changes in the same second doing non-intuitive things. scottven: Um, I'm talking about the "attribute change disappears because it's subsumed by one with a timestamp in the future". scottven: Yeah, same thing. . o O ( Where "future" includes "now" ) xorian: I thought that would be more complicated to fix. Are you just going to make the timestamp be max(now, current_highest_timestamp + 1)? I thought I remembered the replication issues being too great with that one. scottven: Something like that, yeah. I'd prefer to make it max(now, current-higest-timestamp-with-same-name +1), which is part of the compliaction. scottven: Replication shouldn't be a problem. This change would only take effect when the default timestamp is used for an attribute write operation, which normally happens for all local attribute changes (though a client can explicitly specify a specific timestamp). scottven: Attributes copied via replication keep the timestamp they were given when set at the remote repository. The two attribute histories just get merged. xorian: I'm just trying to think if there are any suprises that can happen by merging in a timestamp from the future. but I guess as long as the local updates don't explicitly as for "now" as their time, they'll get future-time + 1 so everything will work as expected. scottven: We could go hold a panel on it at that time travellers convention! :) xorian: yes! scottven: Right, that's the idea. is waiting for Penfold to get back online since about to do some seriously nasty stuff to our main live server on-the-fly "For sources this ... For deriveds, it would be whenever the tool finished writing it." ...but if a dervied has not changed, how will it preserve the mtime? It *may* be in the cache - but may not be :( "what you mean by "attribute programming"" ...we've built a complete 3-tier deployment system subsuming vesta by doing programming in the attribs; using them for holding FSM state overlaid over our files etc ...and it's clear you can do a LOT in that direction. Basically, it's become more clear to me that attribs as implemented in vesta provide a surprisingly ...easy-to-use programming layer to implement apps from within vesta. Sort-of. PS: the description for vesta on the list of SCM's on berlios.de still says "Replaces both Make and CVS, and so can only be built with itself for the time being." (although the longer desc on the vesta-only page is updated) "What would the wiki page about "vcheckout -F" say" ...that it's really not a good idea, and you don't want to use it. Unless you really really know what you're doing? ::) "About the HTTP encoding-negotiation" ...http://www.w3.org/Protocols/rfc2616/rfc2616-sec12.html and (links to) http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1 ...Accept: text/plain; q=0.5, text/html, text/x-dvi; q=0.8, text/x-c ..." Verbally, this would be interpreted as "text/html and text/x-c are the preferred media types, ..."but if they do not exist, then send the text/x-dvi entity, and if that does not exist, send the text/plain entity." ...essentially, client and server say "I'd like X with preference [0.0 - 1.0]" and whichever chooses (server in HTTP) combines the floats then does a sort ..."I'd prefer to not turn this into something that takes multiple round-trips" - ah, yes, see what you mean. HTTP is designed to do that too "I've actually been thinking for a while that we should try to sell more "serious"" ...sounds a good idea. But...this is the thing about marketing: need to strategise (just like an SRS / spec) what you can do that will have most effect etc ...it's really just engineering in disguise :). ...So, for instance, might decide that getting a couple of "serious" disciplines as poster children would be easy to leverage into lots of interest from wider ...community, along the lines, "look [ignorant person too lazy to properly evaluate your software], *this* SCM is used by "lots" [multiple unrelated industries] ... of people who are bigger and more careful than you, so you can probably be lazy and just say "good enough for them, good enough for me"" ...which then also gives you the basis for deciding who to target: you have a definite aim in mind, and can be more focussed and more efficient in getting it "there's only so much you can do with the attribute name...hammer everything looks like a nail" ...yup. Exactly my fear. "But what alternative would you propose?" xorian: hang on, that HTTP RFC doc quote is WRONG (!) AFAICS. It actually means, in english, "text/x-c, then dvi or html, then plain" ...to start with, make a wiki page and gather use-cases on the current attribs, and list and categorize how we're actually using them, and brainstorm how we might wish to ...which then gives us a basis for taking a step back and saying "Right. What are we actually evolving this towards, accidentally?" ...and might throw out some trivial suggestions for ways to go forwards without creating / encouraging a big heaving mess. ...at the VERY least, attribs desperately need namespacing! c.f. http://wiki.vestasys.org/VestaAttributes#preview (IMHO) adammartin: "but if a dervied has not changed, how will it preserve the mtime? It *may* be in the cache - but may not be" adammartin: Yes, but I'm not sure we can do a lot about that. I mean, we *could* do what I said before (compare the existing file to what we would be copying over it, and don't copy unless it's different). But that still seems like a lot of expense to me. weeeell...as it stands, with the whole "you get .log files whether you want them or no", is this kind of thing not the purpose of them? (not that that sounds a robust route) be cheap - only copy if cksum $file differs Maybe it's worthy of an RFE, but you could do that with a script too. (Ship some temp directory, and then compare the files and copy the ones which differ.) that'll catch you 98%+ of the cases Penfold: Rigth, but computing the checksum would be expensive. It requires reading both files. And you could actually do slightly better by just comparing the files rather than checksumming them. Then you could stop as soon as you hit a byte that's different, rather than definitely reading the entirety of both files. mm: but if you compute a checksum every time you put a file in teh repos... adammartin: Well, I would say that the .log files are really meant as an annotation for users. They're not really designed for this purpose. they're a pain in teh arse :) Penfold: But you don't. We do something like that (fingerprinting), but only for files below a certain size. The current .log file would tell you when that file was last written by the evaluator and which shortid was written there. But, it wouldn't tell you whether it had been modified since then. And, making use of this would require writing a .log parse (ick). can you solve that by shipping as links? xorian: or perhaps writing logs in a less unparseable format? :) I guess, if you were parsing the .log file, you could compare the ship time for a file to its mtime and compare the shortid to be shipped to the one shipped last time. If both match, then don't copy. But that's not bullet-proof (mtimes can be changed) and it doesn't deal with Adam's "what if it's not in the cache" case, as shortids change when you rebuild something that has fallen out of the cache. scottven: Hmm, I suppose, but if you ship as links it seems like you've already avoided the entire issue. xorian: I'm not seeing how that's bad. :) Penfold: You've got to remember, Vesta has grown out of a research project. I'm sure the developers never considered using .log files for anything more than an annotation for humans to read. adammartin: So why not ship as links? (Not that I doubt that there's a good reason.) "ship as links" ? goes to check man page to see if he's not noticed something there adammartin: The "-s" flag. It creates symlinks rather than copying files. (is dealing with an urgent problem; afk for a bit) adammartin: The major downside is that if anything you've symlinked to gets weeded out of the cache, your links will become invalid. adammartin: HTTP encoding-negotiation: I still don't see a big difference. adammartin: It uses floating point values for priorities versus integers (positions in the client-prioritized list), but that seems like a small difference. adammartin: When you say "attribs desperately need namespacing", what does that mean? I mean, you could use attributes with names like "namespace:name", or even "ns1:ns2/ns3;ns4#name". adammartin: It sounds to me like you guys may be the foremost experts on "attribute programming". :) adammartin: So feel free to wiki some stuff up, and I'll try to get Intel people to join in the discussion. xorian: re ship/mtime comparison - no, it's not bulletproof. But copying *more* than you have to is merely a minor pain Penfold: Well, let's say there are a number of different algorithms one could use to decide to not copy some files when shipping. It's not clear to me that *any* of them is a good idea. Whereas setting the mtime of the shipped file to be the same as the shortid copied seems like it should always be safe. It's imperfect, but it doesn't seem less so than the other choices we've discussed. Of course if someone were to, say, submit a patch that implemented one or more of these ideas, I'd be happy to work with them to incorporate their changes into Vesta. :) re: htttp encoding algo: I'm just a fan of re-using protocols if they fit; I wasn't really comparing it to what you were suggesting re: namespacing: Damn C programmers ;P. Yes, you can namespace everything so long as you have practically infnite strings (BTW: any length limit on attrib vals?) ...but especially worrying is that there's no namespacing on the attribs that vesta uses, so you could VERY easily conflict. Some of them are things that adammartin: No fixed limit that I know of. ...I've already wanted to re-used and carefully checked and stopped myself in time (e.g. "package-parent" has lots of meanings other than vesta's) (even "package" has mulitple parallell meanings - vesta pkg? deb pkg? etc ...) I should mention that the repository keeps all attributes in memory. So if you start sticking really large stuff in there, you might need to buy more RAM. :) adammartin: Yes, the domain of software has big problems with naming. It's because it's all this ephemeral thought stuff. We just pick random words that seem to fit. last thought before I leave: vesta support for namespacing would be brilliant; just basic things to let you work in an assumed namespace and stuff like that adammartin: I agree that it would probably be nice if there had been some namespace convention and if all the Vesta tools used something like that. It's a little late now though, unless we want to write something to fix up all the attributes in your entire repository. Will comment more tomorrow when I get in. Got train to catch! Bye all ... (thinks: surely iterating across all attribs would be pretty trivial to implement?) adammartin: Yes, it's definitely do-able, but then the upgrade path past the point where we change the names of all the attributes becomes annoying. I would instead advocate that people jsut start using some prfix for their own application-specific attributes. ooo, gee.. like, say "wms-" nah... we didn't do that </grin> Penfold: If it were me, I'd choose a separator other than '-', as that's already used all over the place. . o O ( vattrib -a "This is a funny attribute name" "And this is ts value" ) yeah, let's say that all attribute names that don't contain a colon are owned by Vesta. so user names have to be of the form namespace:name. :-> or pick some other character instead of : some of adam's remarks about attribs puzzle me, since they are just data, not programs. "attribute programming?" "like putting a perl interpreter on every field in a data-collecting app and allowing anyone to type anything and evaluating it before recording it" but attribs don't get interpreted. they just sit there. mann: Yeah, I was a little confused too, but something he said later ("using them for holding FSM state") seemed to clear it up a bit for me. So I think he means using them to hold a significant amount of state which drives some other process. Which isn't too different from some things we've done at Intel, though I get the impression they've gone further. Actually, at both Compaq & Intel. "Client exiting" evening ehjay_home: Hi. looks like you had some good discussions on revctrl ehjay_home: Yes, it's been interesting. ehjay_home: So you like Darcs' patch algebra, right? Have you actually used Darcs? no, never used darcs I just figured it's merge capability was similar to clearcase someone on #revctrl thought that vesta was similar to clearcase. it is in the sense that builds are done in it 's own filesystem it's just that the user interacts with that filesystem as a sandbox ehjay: why don't you write a vesta vs. clearcase page in the wiki? I don't feel like I know enough real details to do so ehjay: AFAIK, you're the only person to have ever used both. and it's been a long time since I used vesta..hardly remember it. besides, it's a wiki page... it can be cleaned up if there are inaccuracies. I wonder if it's possible to export the vesta chroot to a user.... scottven: Ralph Zajac has used both, but he claims he doesn't remember enough about ClearCase. I understand more about the usage model than the implementation of clearcase ehjay: there ya go! you make an initial cut at the page, and we'll get Ralph to come and clean it up. usage model is fine. I'd really want to use vesta again before I did that. Maybe I can do a write up of how the the clearcase usage model works for your guys benefits. ehjay_home: You can apt-get Vesta you know. :) must not be in sarge... I have to add a server to the sources list ? ehjay_home: Yes: "deb http://pub.vestasys.org/debian testing contrib" when you install vesta, do you have to seed it with some initial operating system? ehjay_home: Then "apt-get update ; apt-get install vesta-common vesta-server" ehjay_home: You can replicate everything you need from the rpeosiotry on pub.vestasys.org ehjay_home: See also: http://www.vestasys.org/doc/getting-started-installable.html either of you know of any irc<=>http proxies/tunnels? ehjay: as in something so you can irc from work? exactly there's got to be an irc to http encapsulator out there somewhere I don't know of one. ehjay_home: You guys can't ssh out? You don't have a SOCKs proxy? yeah, but I really hate curses based irc clients ehjay_home: Then use port forwarding. don't have it at work no socks servers i was using a http tunneler to forward ports but they caught me :) ehjay_home: I meant ssh port forwarding. ehjay_home: http://cgiirc.sourceforge.net/ you can't forward over ssh if there is no route to the internet at sun we had a socks proxy which handled ssh, and I could port forward through that. ehjay_home: So you *can't* ssh out from inside? only way to ssh out is to a ssh into a severely restricted landing pad, and then ssh out from there. of course they have forwarding disabled in the ssh client on the landing pad ehjay_home: Ah, now I see. ehjay_home: Anyway, why don't you try cgiirc? It was the "I'm feeling lucky" hit when I Googled for "web-based irc client". yeah, I ran into that...I might try it. . o O ( http://www.nonzerochance.com/index.php?p=buypage&design=damnsearchengine&style=tshirt ) get the shirt and I'll stop :) later, off to bed Just appeared in vlatest - http://pub.vestasys.org/ : /vesta/vestasys.org/vesta/extras/pkg2vesta/5: Added --from-installed (copying the files of an inctalled package) support for RPM. Also added messages for vcreate/vbranch/vcheckout to avoid querying [...]