I'm disturbed to realize that darcs's Achille's Heel -- nested merger lockup -- is hit by long-lived branches as well as by doppleganger patches. what do you mean with the former? when you do not exchange patches between two repos/branches for a long time? I mean repeated merge. I have some cleanup patches on a branch of mine. My boss Jim continues to change that code. Whenever there is a conflict I merge his latest changes with my cleanups. But now darcs has begun locking up. Very disappointing. yep Which distributed SCms maintain local branches? olczyk: Mercurial and Codeville can, surely others, too olczyk: What do you mean by "maintain"? Do you mean "allow"? surely you don't mean "force to remain local"? MT has a strong local/remote distinction, too. ah, local == private? Basically I mean that a single repository contains branches, that you can check out two different branches from the same repository, and that you can tell there was a branch by examining the history. olczyk: then my first answer is correct. Both call them heads olczyk: Vesta can do that too. For example, darcs does not support local branches. Only global ones. Vesta is distributed? olczyk: Yes. At Intel we use it at sites on both the east and west coasts of the US and in India. They all inter-operate quite happily and seamlessly with local operations taking place at each and eplication between. http://wiki.vestasys.org/VestaFAQ/GeneralQuestions#distributed s/eplication/replication/ So these are systems in which branches that want to interact must be in the same location? And then there are tools to add data from other locations? abentley: If they're never in the same location, how could they interact? I mean, you can't merge with something you don't have. Hmm. The way Vesta talks about branches and the way I do local and global are exchanged. xorian: By patches. abentley: uhm, same location? This doesn't sound very distributed. What do you mean? abentley: If that question was specifically about Vesta, then yes, you can have branches that are local to any of those three and which then get replicated to one of the others on request/demand. In one case the SCM allows interaction through commands (my local) and in the other allows interactions through some sort of patch system ( my global). ThomasAH: In these systems, everyone has their own copy of the project, and in order to merge you have to pull the remote data first. olczyk: OK, sure, but you still need to move the patches. And to have a reasonable chance of correctly applying them, you probably need the history of the other branch too. abentley: yes, though Mercurial allows merging completely unrelated repositories, too abentley: but after this the repo will contain both complete histories So it is distributed, but not as aggressively as, say Arch or bzr. In Arch, you can't pull another archive's data into your own (you just mirror). In Bzr, you can, but you don't have to. olczyk: I don't see how command maps to local and patch maps to global. I mean, a command is some operation that manipulates some state the VCS is storing for you. Whether or not that state remains local or gets propagated somewhere else seems orthogonal abentley: Mercurial will warn you if you do this, so you have one chance to use the undo feature :) abentley: In Vesta you can replicate as much or as little as you like. The objects that you replicate are immutable versions. I mean a system where you have a type of branch which can interact with other branches must be in the same location and other branches which can interact through tools. abentley: Close but I don't like the "must". Sorry got those two lines backwards. abentley: Also, a Vesta repository holds multiple "packages", and each package is a separately version/branchable thing. So you could replicate someone else's unrelated package intoyour repository and have a copy of it, and this would have no effect on the versions of your package. Now I know why I pased on Vespa. No WIndows. olczyk: Yeah, none of the contributors do Windows :/ xorian: this has good sides, too :) And also it would be challenging to get Vesta's big win (complete versioning of every file used by a build) on Windows, as the UNIX method of chroot-ing into a temporary filesystem isn't an option. So let me pick an example of a project, the Linux kernel. So you create two "my local" branches, 2.4 bugfixes and 2.6 development. But some guy wants to write a new driver. So he creates a "my global" branch off of 2.6 to build the driver. xorian: So that would include system headers? you want in either 2.4 bugfixes and 2.6 development, visible to everyone. abentley: Yup. Builds can only use wahat's in the chroot, and that's populated exclusively from what's under revision control in the repository. Sorry. The official changes to the kernel, you want in either 2.4 bugfixes and 2.6 development, visible to everyone. Until the driver is near release you don't want eveyone to know about the changes. The history of 2.4 bugfixes and 2.6 development are in the same repository, but the history of the driver is not. What I weant to know is which SCMs support the first type of branch? All distributed scms. You just use a second repository for private stuff. Private stuff that may in the future become public. olczyk: Yeah, this sounds pretty simple. You're just saying that you have private "2.4 bugfixes" and "2.6 development" branches which are separate from the public ones and may contain stuff you don't want published. I'd be surprised if any modern system didn't support that. Well let me give you an example of an SCM that doesn't: darcs. To branch in darcs you create a copy of the repository. olczyk: Not even with a separate Darcs repository? I mean, isn't that the way Darcs manages branches, by reating a new repository? So there is never a public record of a branch. Even if you want one. Exactly and for that reason no repository can ever know of any branches. In darcs a history is always mostly linear. olczyk: Well... why do you need a public record of a branch whose contents aren't made public? Incidentally, Vesta *can* do exactly that: publicize the existence of a branch which you only selectively publicize certain version of. The changes appear in a line. xorian: That's the point. Sometimes you want branches that are public and sometimes you want branches that are private. Darcs only supports private branches. BVecause in darcs there is a one to one relationship between branches and repositories. In CVS you have branches within the same repository. CVS only supports public branches. olczyk: Well, not that I'm a Darcs user, but to make a public branch don't you just publicize you branched repository? How? olczyk: Um... however you make a Darcs repository available to others over the net? <- still not a Darcs user :) The is a one ot one correspondence between branches and repositories, so you have to make two repositories available. Plus some description. olczyk: So, what you want is an index of the different public/private branches that's publicly accessible? In other words all the mechanisms for handling branches in CVS you have to implement by yourself in darcs. zooko: Wake up already Not quite. Imagine a graphics program that takes the history of a tree and draws a tree. In CVS that history will look like a tree. In Darcs it will look like a straight line. In darcs any branch will be "invisible" with no possibility of making it visible. olczyk: Other than having an index of branches, how does that matter in practice? I mean, you could reconstruct that tree by finding the common parts of the histories in the different repositories, and I'm sure Darcs must do something like that in its patch algebra engine. In practice, In CVS I am working on bugfixes for 2.4: command "cvs -branch '2.4bugfixes'", for 2.6 development "cvs -branch '2.6development'" olczyk: When the private stuff becomes public, you pull it into your public repository. In darcs, "darcs url of 2.4bugfixes" and "dars url of 2.6 development". If I forget what the branches are in CVS I ask for a history, in darcs I have to look through my notes. Darcs does not have the machinery for handling the branches, I have to provide it. abentley: Exactly. In a distributed SCM you cannot avoid private branches. Because private branches are just mirrors of your repository. So bzr also doesn't have a built-in branch directory service (atm), and Arch does. I'm not that familiar with bzr. I believe arch does, unless I am misreading documents. olczyk: If you wanted to avoid private branches, you just ensure that the repository is public. There may be bandwith considerations, but there's no technical barrier. I like the "graphical history" analogy because that makes it clear. A SCM with a "branch directory service" will have a tree for it's history. One without will have a line. olczyk: This is not true. the directory service provides the locations of branches. But without the directory service, you can still see a branching history, you just can't access the original branches. abentley: But there is no way to prevent a preson from replicating your repository. Which is a way of creating branches ( ala darcs). Only if you require that replications are recorded in the history. True, no distributed SCM prevents people from branching from public repositories. replication doesn't necessarily mean branching. In essence, I'm looking for a SCM that can hold two branches in the same repository. xorian: How can replication not mean branching? olczyk: That still seems more like nomenclature than functionality. But Vesta can hold innumerable branches in a single repository! :) olczyk: What if you replicate but never modify locally? Like you just want to get a read-only copy of the source and its history? Maybe for auditing or building purposes. olczyk: you could take a SCM that holds one branch in each repository, and then "mkdir orepo ; mv darcsrepo1 orepo ; mv darcsrepo2 orepo ; cd orepo ; echo "darcsrepo1 contains the this and that as of now" > darcsrepo1.txt ; echo "darcsrepo2 contains so-and-so's such-and-such" > darcsrepo2.txt". zokoo: In other words you can use use an SCM that holds one branch in each repoitory, and then replicate all the machinery by hand that an SCM that holds multiple branches in one repository gives you. What is "all the machinery"? The suggestion I gave was a .txt file per branch. What else do you want? I've been seriously considering something like this myself, as I use darcs, and I find myself putting long directory names. wrote a silly tool to simplify this kindothings :) Here are the names of a set of darcs repos that are in a certain directory on my system: beta1 d2s d2sb dw dw1182 dwex findnestedmergers justzookopatches like-beta1 subset tailorizer tailorizerd2s working sometimes I forget what meaning I have encoded into them. "dwex"? Was that "darcs world excluding" something? I forget. So I've been thinking maybe I should take notes. Although in practice what I do is throw away all but seven plus or minus two of my darcs repos. :-) lelix: what tool is that? it's incomplete, but basically "literate repositoring" :-) using ReST as a storage system :-) zooko: It sounds to me like you're making his case for him about the user having to do the book-keeping. zooko: I don't want to specifiy "all the machinery " for two reasons, 1) Because I will forget something, 2) because I won't specify a set of orthogonal actions. Well, I am agreeing with olczyk that something more is needed. Although (a) it isn't needed so strongly that I've actually bothered to "vi notes.txt" or to scribble on a piece of paper, and (b) I'm not sure what would be useful beyond a couple of hundred bytes of scratch space to write notes about a repo. There might be some really useful things that a tool could do by managing multiple darcs repos (i.e. managing multiple branches). I'd like to think of some. I want something to tell me what branch I am working on. I want the ability to create new branches with one command. I want to be able to switch branches seamlessly. I want to be able to create a list of branches. Note that currently I *am* using a couple dozen bytes of scratch space by encoding things into my directory names. olczyk: all of the four things you just stated already work for me when I use darcs. What branch am I working on? pwd -> /mnt/hdc1/zooko/work/newnewtailorfrom770/findnestedmergers/working-subset1 create new branch with one command: "darcs get working-subset1 working-subset1-newbranch" switch branches seamlessly: "cd ../otherbranch" create a list of branches: "ls" olczyk, what you should really explain is: what actions do you wanna perform interbranch? I agree with you that these are absolutely necessary actions to have if you are going to be using a multibranch system. sharing patches? obtaining diffs? Here are some interbranch things that can be done only painfully with current darcs: I also want the ability to noninvasively find out some of the new branches I create. 1. get a list of patches which *are* in A and are *not* in B this can be done only with manual effort or bizarre bash scripts Although I guess I have to admit that I've used a bizarre perl script that does exactly this by interactively poking at the text-based interactive darcs UI... 2. Tell darcs: "pull all patches from that branch to this one" (darcs already does this) "... EXCEPT for any patches that are over in that one" (darcs doesn't currently do this without aforementioned hassle) zooko, what do you mean with "that are over in that one"? 3. Um.. Actually I can't think of any more at the moment. lelix: I mean I have patches in A, B, and C, and I want to pull everything from A to B *except* for patches that are also in C. oh I want to create a clean audit trail of all the publicly announced branches. This is a frequently requested issue nowadays. olczyk: I haven't understood what you meant by "find out some of the new branches I create". yes, I see olczyk: what do you mean by "create a clean audit trail"? is going home, will follow from there later all Back to my kernel example: I'm Linus in hiding. We've just created 2.4bugfixes and 2.6development. I want both of these branches to be publicly announced. For "create audit trails", Linux has just died. I am a manager in RedHat. I hire Deloitte and Touche to examine the code history and tell me what the public branches are and what the status of each is. Where the branches were created. What exactly they incorporate. If there has been interaction between the two. olczyk: In a distributed system, you can only do that if people cooperate. Or better yet an example where I would use such an audit history: I work for Lucent, We are working on 5ESS Sri Lanka, Version 6 SR3. Sri Lanka has just passed telcom laws requiring the same kind of functionality in my version of 5ESS as in India Version 4 SR7. I want to find and incorporate all those changes between that and India Version4 SR6. abentley: that is true, but i would submit that people are not looking for truely distributed SCMs. Just semidistributed ones. In fact, you can't even do that with CVS, because people make their own clones of the repository every now and then. Hell, canonical is importing projects from CVS to Arch all the time, without needing the projects' consent. Can you name one serious project that doesn't have a "main " repository? probably not. But there's no way you can prevent people from copying a public repository and doing their own thing with it, no matter what the SCM. Well with CVS it would be hard to grab the whole history if you don't have the repoitory :) If the repository's public, you have it. Only if you have access to the filesystem. Canonical doesn't need access to the filesystem for their imports. yes but it's hard to grab the whole history if you don't. Besides I'm not saying that I want to stop people from copying a repoitory. it's fair to say that semi-distributed systems might be more what users want. unfortunately they're not quite what most of us have gone off and *built* :) perhaps we've built what users don't want. time will tell. What I am saying is that on any project, there is the concept of a "main" repository and people want "official" branches and the machinery to mainipulate those branches. Some people don't want "unofficial" branches. They're called managers. I agree with both these points. I also wish to point out that neither of them featured prominently in the design of some (all?) of the distributed systems recently assembled. Others do, and in a distributed or semi distributed system it is impossible to prevent it. some people want these things. perhaps some of them will be let down, when evaluating each candidate system. there are many tasks VC systems are designed for: communication, synchronization, record-keeping, resource-controlling, simple backups... olczyk: it still appears to me that the things that you want to do are fairly easy to do with darcs. Perhaps I don't appreciate some particular subleties about it. Sigh. So much to discuss and I have to go... :-) zooko: Well for one thing, can you gaurantee that each person on a project does it the same way? olczyk: regardless of the revision control software that you are using, to guarantee that all the participants conform to certain behavior requires management, contract, peer pressure, police, or something else. I know that sounds trite. But I honestly don't see how your objection applies to darcs. By the way, I, like Canonical, have been slurping entire histories of open source projects from their CVS servers, without their permission or knowledge. Do you trust that idiot who when using RCS would go on vacation and leaves his file locked is going to create a branch adhering to the prescribed protocol? s/file/files/ thinks. I think I understand your objection. The thing about cooperation with strangers in the world today is that using a tool which constrains you in a certain way doesn't constrain *them*. I can name numerous widely used open source projects which currently use CVS or else SVN. Perhaps they do so in part because they wish to deter people from making branches without their knowledge or consent, to prevent people from forking willy nilly, etc. Or more to the point, can you guarantee that the guy who wants to unlock his files but gets distracted by something and forgets, is going to type the same three commands in the correct order each time, or is he more likely to type the one command? However, I and others *have* done so -- have created branches willy nilly without their consent, using tools like tailor. So I guess the point isn't that you need to choose a tool that doesn't facilitate this, and give that tool to your developers, but in addition you also have to make sure that your developers don't find tailor.py and start using it. Which is easy if they are under employment contract to you and work in the same building and all that. But in that case, the easy case, then you might be able to give them the more powerful tool while also managing their use of it. But I agree that this is a grey area. Here's my menagerie of open source projects which I have unilaterally and cruelly forked without their permission: https://yumyum.zooko.com:19144/cgi-bin/darcs.cgi for most of them I've made little or no change to their actual source code. For at least one I've extensively updated it for my own amusement. (crawl) I believe really you are talking about somewhat fundamentally-different families of work when you consider open vs. non-open source programming. free stuff embraces copying. copying is a virtue; we try to find ways of making ourselves powerful through use of copying. the natural corollary of copying is "divergence of the copies". tools founded from this stance will have a different set of things they're good and bad at than things founded from the stance that copying must be controlled, minimized, or eliminated. (not making a moral judgement on either stance mind you; I'd be happy if we could control copies of flu viruses for example...) graydon, that's not what I said. In fact I just said what you are saying. But I am also saying that at times everyone wants some control. If they didn't SCMs would not exist. olczyk: some people want control over their own copies, their own histories, etc. Others want to control the way other people use their copies, etc. Like Graydon, I'm not saying that the latter is necessarily bad. It obviously has a lot of use, for example in commercial software development. "that's not what I said" <-- what's not what you said? I didn't think I was attributing any stance or statement to you. Which I do, by the way. so do I! I even do development where we keep some stuff secret. and other stuff which we need to keep 10-years+ of bit-for-bit-reproducible history What I'm saying isn't that you should want to lose control over other people's use of their copies. What I'm saying is that if you want that control, you aren't going to get it solely by using a revision control tool which facilitates it. Like I said there are two types of guys. The guy who leaves his files locked and doesn't, and the guy who wants to unlock his files but just as he is going to unlock them another developer comes over with a question about the design and he forgets. That developer wants a tool that will help him remember. (in the latter case my recommendation is to archive the entire system state, plus a hardware simiulator for the known-runnable state, and make dozens of copies of it all over the planet, and forget about VC tools *entirely*) That developer doesn't mind doing three things in the same way to create a branch-- he probably agreed when they discussed it. But he would rather have one command. graydon: have you gotten the Vesta spiel from xorian or from other sources? I like it. (re: bit-for-bit history) olczyk: I strongly agree that minimal "hassle" is an important value. I agree that a first-class "branch" formalism is helpful, simply for *communicating* your intention; I disagree that most of these tools will actually enforce the formalism. they might, at best, "remind you of it" (in your terms). monotone will remind you of the notion of a branch, for example. but you're free to ignore what it says. If I have to type two separate things when every time I do it I only mean a single thing, then that bothers me. zooko: I'm familiar with vesta, if that's what you mean; I'm not sure if the xorian-provided spiel has any special parts I would not have gleaned from the documentation. Okay. Well I've got to go now. Bye Bye olczyk! zooko: I did actually recommend we use vesta for the long-term-archival needs. and this suggestion was ignored, alas, like every context I've ever suggested either aegis or vesta for. Hm. people don't accept process suggestions unless they come from above. it's weird. it's like nobody's aware of the improvement in systemic function when everyone uses a codified and predictable process (*any* process) Perhaps there's a coordination issue. No sense in adopting vesta unless all of your co-workers are doing so. It's hard to get everyone to agree on when and how to do such a thing. Also to decide on where to go for lunch. If it comes from above, everyone knows that everyone else will do it, so each is happy to do it. yes, of course. I don't mean it's "bad", just noteworthy. there's also a curious unix tradition of respecting a tool-preference boundary for each user. perhaps this is actually at the root of the "small tools" aesthetic: the sense in which each hacker's unique pickyness can be accommodated by letting them write their own (or choose their own) idiosyncratic tool. the need to communicate seems to be the only thing forcing users to use the same VC tool, otherwise I suspect they'd all manually implement something like quilt and rsync-backup.. it's especially odd considering that it's so much cheaper to force users to adapt than to produce a program which adapts to users in their natural state I guess economics is against it though. it's cheaper only when you can control the users; when they have choice and disposable income they pay a premium to have the program adapt to them, and the zero-cost-per-unit makes this a plausible way of making money writing intuitive programs. graydon: interestingly, I was wondering yesterday about the feasability of other VCSes as 'foreign branches' in bzr, such that you could merge from them as they were updated. Contributing work back would be a problem, though. we have an experimental monotone branch which does this. tracking a CVS branch as external to monotone, pushing and pulling versions back and forth. I've not tried it. the author says it works. I wouldn't think CVS held enough state for that to be useful. many people prefer using CVS, or like CVS as a "backup" format, or want to keep using CVS while tinkering with new systems.. We certainly didn't want to drop CVS before we were confident in ARch. graydon, abentley: I've been doing a *lot* of that kind of stuff with the tailor tool. So far darcs has always been one of the sides. The other side has always been either CVS or SVN. It's not perfect, but it works, and it appears to me that tailor is destined to grow more and more useful and to interop with more and more revision control tools. ^-- the "only darcs, only SVN/CVS" stuff is my personal use. tailor already supports far more than those 3. Do you guys thing a unified changeset format is desirable or even possible? s/thing/think The big question is what information goes in. The small question is how to encode the information. darcs requires strictly more information than codeville-new/monotone-new which requires strictly more information than arch/monotone-old/etc.etc which requires strictly more information than cvs/svn, IIUC. zooko: I agree, that's the big question. Would SCMs that didn't need/want the extra info be willing to go along with it, and would they support the more demanding SCMs? I' I'm not sure we even agree on what changesets are for. In bzr, they're for transmitting a revision cheaply. So *currently* as far as I understand it darcs requires *live* interactive access between the source and target darcs repos. what do you mean by "changeset" abentley? what is the ... uh... ontology of revision control implicit in that? In fact, the current monotone does as well, AFAIK. So there is no way that those two could currently export a single static object that another system could reliably interpret, AFAIU. tomlord: well, I personally think of changesets the way Arch 1.0 had them. zooko: after seeing git, i've started to think that "repository" shouldn't really be part of the revctl world. Sure, there are places where revisions get stored and those can be called repositories --- but I don't like, anymore, the idea that a particular revision "lives in" or is somehow tied to some specific repository. OTOH, tailor.py already has an abstract notion of changeset and it already maps many systems onto it. However, that abstraction is tied to a full local system state -- can't easily be serialized (Except via the obvious kludge of serializing the full local repo and its history. :-)) nods. Though bzr's changesets are designed to be as close to patch as we can make 'em. As close to unified diffs, I mean. tomlord: yes, what you envision is, I guess, exactly the thing that I just said that darcs doesn't have: "exportable", atomic, self-sufficient format. (that's what arch 2.0 has, btw. new release today or this weekend.) monotone's changesets are just a bit of extra context connecting the pre-state manifest to the post-state (noting which of the observable changes meant add, del, or rename) i'm not sure rename has any proper role there. it's important for some cases of merging, sure, but that's different from just shipping around the trees. we didn't think we'd need this context at first, but tomlord was right, it is too ugly (or you lose too much of the user's intent) without. well, it has no role if you have some sort of file GUIDs. since we don't, it has a role! that's funny. it sounds like we switched sides, partly by convincing each other that the other was right in the first place :-) (rename tracking, that is) Oh, so I am wrong about monotone. tomlord: I like "patch pools", btw. Centralized collections of revisions with no directory service. tomlord: yeah, probably something like that happened. you know, grass being greener at a distance :) zooko: wrong about it? how so? linus found the low note: it's a distributed filename first and everything else layered on that. I was wrong to think that monotone requires a live connection to another live monotone in order to exchange a patch. darcs does. live connection? In order for a darcs executable to know how to apply a patch P to a repo R, where that patch P is currently in a repo S, the darcs executable needs to examine (parts of) both R and S. If R and S are on separate machines, it requires a darcs executable running on each machine and talking to the other one. tomlord: perhaps. I'm still not sure whether there's a single perfect substrate for VC systems. there are always questions about identity and copying; what to unify and what to allow to diverge; I don't think any of us -- nor git -- got it "right" or "wrong" universally zooko: Could a person creating changeset P' determine in advance what would be needed to apply it to S? Err, R? zooko: ah. there's no facility in monotone to apply a patch without its ancestor on hand, at the moment, at all. you would need to have fetched it first. graydon: In bzr, the changeset-basis can vary-- using the direct ancestor as a changeset basis is just a convention. it's not just that you have to *examine* the predecessor; you have to have the predecessor. the database doesn't let you write revisions without their predecessors. you could detach a changeset from the revision and apply it in a new context, but we currently have no commands which deal in changesets independently of revisions. graydon: identity (if i take your meaning right) should be a user-assigned name plus a very hard to forge signature, in a system where *usually* just the user-name, as a short-hand, is unambiguous enough. linus didn't get that quite right and didn't think hard enough about distribution. graydon: Ah. We require that the changeset-basis already be stored in the target branch. graydon: okay, so I am sort of right about the current state of monotone. Because for us, the sole purpose of a changeset is to reproduce a revision. But darcs is even a step more complicated, in that it doesn't know in advance which parts of the source repo and the target repo it is going to have to examine. abentley: arch family is f'ed up because it conflates the logical history of the tree with the set of prerequisits for building a given revision -- the thing that archive-caching etc. tries to compensate for. There's no inexact patching, but you can merge a changeset. abentley: actually it could in theory, there is even a little practice in which people do that. But it isn't the main stream yet. could determine in advance the context state necessary for transmit the patch, that is. tomlord: I can't really make strong statements about what "should" be. I don't really know. the way users turn out to like or dislike things, and the way tools turn out to be stable or fragile, has surprised me too much to take such clear positions. tomlord: You don't have to tell me, after our endless skip-delta discussions! graydon: i know what you mean and often feel that way too but to embrace that too far is to throw up ones arms and just give up. one has to risk making informed guesses at some point. abentley: maybe i'm just reporting that "i finally get" what you and others were pointing at. tomlord: of course. but my informed guesses are, well, I find most usefully expressed in working code rather than english. I often find I only discover what I mean by a statement when I try to implement it; and I only discover what's wrong with it then as well :) tomlord: Cool. Speaking as not-a-developer but only a user of these systems, my primary current desire is that someone else learns from darcs so that I can have more than one system in the darcs space. I don't mean to say that others haven't already "learned from" darcs. But there are currently no others available that have the same commutation magic. Perhaps cdv/monotone-new-merge-algorithm will be a good alternative. mpool refereced Darcs a lot when we were first talking about bzr. it doesn't really disturb me any more that we've got 20 or so systems in this space; it seems actually to be a natural product of a bunch of people thinking through technical problems which are too subtle to be worked out in english. you need reference code to be able to see when something does or does not work I think a lot of people see loss of history as too great a price for any merging improvements. abentley: I noticed that! It was encouraging, but it turns out bzr didn't decide to go for the whole commutation magic, which is quite reasonable for a conservative design. graydon: (re working code) you can see my latest tonight or by monday at the latest. (release bundling/publication is way too tedious :-) abentley: I think this is an unfortunate myth. Darcs does not inherently lose history any more than the other distributed systems do. I said "inherently" meaning the fundamental concept of commutation/etc., which another system might wish to use. The current implementation and the way it is currently used is somewhat susceptible to history loss, but the prevalence of that is probably overestimated by people who haven't tried it. My understanding was that commutation necessarily involved rewriting history. I'm happy to be corrected. tomlord: Looking forward to arch 2.0! abentley: consider a minimal example of commutation: there are two repos, R and S. in each there is a patch P0 which creates a 100-line file. in R there is a patch P1 which replaces line 99 of the file like this: "-hello world\n+yellow whirled". in S there is a patch P2 which inserts 50 lines into the file like this: "@@ -0,1 +0,50 @@\n-whateverthefirstlinewas\n+... 50 lines" With me so far? Sure. you're going to assert that there's a Right Merge here but that's a stretch. Now commutation in this context means that when you move patch P1 from R to S, that it applies to line 149 of the file. I do assert that this is almost always what the users want, and can almost never be a silent mistake. And that's it. That's the magic of darcs. I think it is unfortunate that this magic is explained in the darcs docs in terms of quantum mechanics and sophisticated algebra -- an explanation which I personally have never yet understood. i dunno, if files diverged that much --- i think the Right Merge might be one that signals a conflict and asks for human review. tomlord: you would be right to think that if the darcs algorithm were using only the current state of R and S. but since the darcs algorithm has more information it can make this inference more reliably. Namely, it knows that no other patches have been recorded in either repository which would have changed the position of line 99. zooko: i'm just a lot more conservative than a lot of the FOSS community about source management. That's why darcs can do this stuff that other systems can't -- it is *not* using a more clever heuristic to make a better guess where things should go -- instead it is using more information than merely the two pieces of "current state here, current state there", and it is also using more information than the three pieces of "current state here, current state there, some particular ancestor". you're getting a little too fancy for my tastes there. i don't believe in magic, i just don't want surprises, and yadda yadda. the main point being that such a merge is a great tool in the toolbox but nothing to architect an entire system around. We don' tomlord: That's ironic-- I was just reading 'undiagnosing subversion', which called you anything but conservative... t use darcs in my workplace yet. The reason we don't is a good one -- the other developers at my workplace do not yet understand how the merge algorithm works. I hope to change this by writing up and posting an explanation. What I wrote above is a rehearsal for that. mumbles unkind words about MIT undergrads in abentley's direction :-) I strongly agree with you that I don't want magic, sophisticated heuristics, etc, etc, in my revision control system. So given the minimal situation that I've described, the options are: so you should give a better example thanthat, because that's the kind of info that is present in other SCMs. 1. Raise it up to the user and ask them to determine where patch P1 should be applied. 2. Do a guess, such as patch's "fuzzy matching" 3. Use the complete revision history to determine where patch P1 should be applied. In this minimal case, #3 -- which darcs does -- consists of observing that there is a patch P2 which inserted 50 lines at the beginning of the file, therefore -- without guessing -- patch P1 clearly should be inserted at line 149 instead of line 99. I strongly agree with you that #2 is a bad idea. #3 is bogus because the "complete history" is about the history of text lines, not the history of what the changes to those lines *means*. taken to an extreme, this is an argument against patching at all, which is obviously too far. i dunno. some kind of balance is needed. tomlord: That's an interesting argument. Can you give me a small example where this approach would go wrong? I can think of one myself, but I'd like to hear others. zooko: I believe the fundamental issue you're complaining about is that most diff3 application involves inferring "line identity" and then synthesizing identity-respecting line-edit history; darcs (and perhaps codeville) record line identity on a commit-by-commit basis and try to preserve the identity when composing long ancestry-graph edges This may actually relate to what Tom was just saying about separating revision construction from history. You can refer to each tree state as a composite of several patches, and then record every tree state through history. graydon: I'm not sure if what you've said accurately reflects my position. I think that approach #2 is to use incomplete information in order to "guess", heuristically, where the patch should be applied. I suspect that many people who rightly oppose automated, silent application of approach #2 are incorrectly believing that darcs does this. Where as, in fact I think darcs solves the problem by using more historical information so that its algorithm is not a "guess". is procrastenating and should get back to work, having muddied the waters here :-) zooko: It's still a guess. It's just a better one. zooko: well, I am refining what you mean by "guess" and "complete history" in cases 2 and 3 I hope that my minimal example can serve as a basic for discussing these things concretely. You need AI to do perfect merging. Applying patch P1 to line 149, *because* of the existence of P2, seems to be qualitatively different than Or mind-reading. any possible algorithm which might decide where to apply patch P1 given only the current states plus an optional ancestor. Does that make sense? But the patches themselves are produced by text analysis. What you guys have said, such as that an AI is needed to figure out where to apply a patch, makes perfect sense to me for any possible algorithm which uses those limited inputs, They're not produced through any understanding of what the changes mean. but does not makes sense to me in this context: where the presence of P2 obviously and uncontroversially implies that patch P1 needs to be applied 50 lines further down. thinks about what abentley said. abently: each patch is produced by text analysis, yes. a question, though, is whether you try to maintain line-identitiy between edges. zooko: when you commit a single patch, a->b, your VC system looks at the pre-state of that patch and the current state and infers a set of line edits. darcs does this too. abentley: can you suggest an example case like my minimal example where it would be wrong for the system to silently translate patch P1 50 lines further down? graydon: newlines are only a hint as to statement boundaries. I can think of one, but I'm not sure if it is an example or a counter-example. ;-) graydon: right. the question is whether the VC system then knits those line edits together, by identifying lines and saying "line A in rev X goes to line B in rev Y", step-by-step zooko: P1 changes the arguments that a function takes. P2 invokes the function using its old signature. abentley: that is a good example. or whether it jumps up a level, finds (ancestor,left,right), and then re-calculates (without respect to history) the supposed line-edit relationships between ancestor->left and ancestor->right It is *not* an example of darcs commutation going wrong, though. It is an example of there being a semantic dependency between patches that textual analysis can't catch. I don't think that darcs patch commutation increases the dangers of those, does it? It increases the dangers of those being committed. That is to say: how does any other system, current or imagined, do better than darcs does on that example? thinks about that. AIUI, Darcs will auto-commit that. abentley: Okay, I see an example of what you are saying. lunch time for me If P1 invokes the function, and P1 applies to line 99 of the file, and P! darn. by graydon. by tomlord bye graydon: see ya. if P1 invokes the function, and P1 applies to line 99 of the file, and P2 changes the signature of the function, and P2 inserts 50 lines at the beginning of the file, then a system like SVN will object that P1 and P2 conflict. Not because of their semantic dependency, which is of course invisible to the revctrl system, but because it doesn't know where P1 should be applied. If P2 changed the signature of the function without changing the number of lines in the file, or if P1 and P2 applied to different files, then svn and, I assume, every other revctrl system in existence, including darcs, zooko: But even if the patches did not conflict, SVN would still treat the results of a merge as an edit. would silently allow the semantically incorrect merge. Good point. For SVN, every merge requires a user to review before committing. And if there were procedures in place to catch incorrect edits, you wouldn't get a bad commit Although in practice this means that users commit merges without actually reviewing them. oh... just thought of something..... i'll bet that assembly language programs provide many excellent examples of the pitfalls of merge algorithms that usually work just great for, say, C. at the opposite end of the spectrum, given how code is formatted, lisp is probably another source of examples. tomlord: interesting. I would welcome any minimal example case you can think of, as fuel for my nascent "darcs magic demystified by example" doc. oh, c'mon.... that's your work to do :-) tomlord: I think this is just a difference in how you break up units. abentley: so as far as I understand from your conversation here, darcs patch commutation does not actually increase the chance of a semantically incorrect merge, except inasmuch as a semantically incorrect merge might also accidentally be a "the location moved" situation. There's nothing especially magic about \n-- it's just that it's conventionally used as a statement separator. The point about SVN is an excellent one: SVN treats every merge as a potential conflict and requires user approval. zooko: It depends on what you consider to be a merge. I consider the human intervention to be part of it. arch and monotone (if I understand correctly...) treat textually "unclean" merges as a potential conflict, but silently process textually clean merges. darcs treats merges which involve conflicting changes as conflicts, but silently processes merges which don't involve conflicting changes. zooko: In Arch, they're still treated as edits-- they're not auto-commited. I see. I do a lot of work in security, and a lesson we have learned is "the boy who cried wolf" lesson. If the alarm goes off every time, it is the same as if the alarm goes off never. In fact, it doesn't need to be anywhere near *every* time before people stop responding to it. The analogy that I wish to draw is that saying "We require the user to manually acknowledge every single merge." doesn't mean that in practice your users are going to inspect every merge. We use SVN in my work, and I have observed that 99% of merges made by any of our four programmers get committed automatically unless there were conflict markers. In contrast, darcs never raises the "conflict alarm" unless there is something which is unambiguously a conflict. Well, my son is calling me to dinner! Thanks very much for the conversation abentley, tomlord, graydon! thank you too, zooko. I thought about it during dinner. The issue of "when to raise the alarm" is separable from the question of commutation. Suppose you work on arch, and suppose arch treats every merge as an edit and requires the user to approve it. Fine. Now in the minimal case above, what edit should arch suggest to the user? I propose that it is clearly better for arch to suggest that P1 be applied to line 149 than to suggest that P1 be applied to line 99. Now I twice offered to give counter-examples and nobody took me up on so I'll do it anyway. ;-) I can think of two counter-examples where the simple commutation described above could be wrong, but both of them are pretty weird. One is sort of the "Go"del" issue: suppose patch P1 replaces line 99 like this: "-hello world\n+if $line != 99: release missile" Then if arch suggests that P1 appear in repository S as an edit to line 149, and if the user commits the suggested edit without examining it, then the missile goes off. The second example is similar: suppose the new line says "If the text 'xyz' doesn't appear within the previous 50 lines of this file, then release the missile". Again, if patch P2 does an "insertion" operation which causes there to be more than 50 lines between "xyz" and line 99, there is a problem. However, I find it hard to think of more "realistic" examples in the realm of software development. Okay, but if we switch from "line number" commutation to "directory structure" commutation then maybe it is a bigger issue. In darcs, a patch that mvs a file from ./subdirA/filename to ./subdirB/newfilename can be automatically merged with a patch that edits the contents of the file. Suppose you have an executable that says "if sys.argv[0] != 'filename': missile.launch()". In reality, executables *do* sometimes inspect their filenames and switch on them, so this one isn't as far-fetched as the others. However, by giving some examples of how it can go wrong, I hope I've challenged you to think of more common examples of how it can go wrong. My current hypothesis is that only these "self-referential" examples can cause the darcs commutation algorithm to suggest bad merges, and that the other 99.9% of merges that working programmers do are better served by darcs commutation than by algorithms which use less of the relevant history. Okay I'll shut up now and I'll write a web page if I want to write so much. :-) Okay, no I won't. I'm still excited about this. Suppose you say that darcs commutation which merges a file-rename with a file-contents-change is dangerous, because of, for example, the "if sys.argv[0] == 'myrightname':" problem. However, for *any* possible merge algorithm, I can construct a Go"delish example in which that merge algorithm will cause the missile to go off. So I think that we either have to accept a small possibility of bad merges or else we have to instruct the user to go off and do the merging herself and come back when she's done! (Which is what Vesta does...) So for practical purposes, what is most important is the trade-off between false alarms and false non-alarms. I submit that darcs patch commutationo *both* reduces the frequency with which the user is required to approve a merge that is "obviously" (to the user) done a certain way, *and* it reduces the frequency with which an unlucky merge can result in a "clean" but semantically wrong state. Along the lines of the simple example above, suppose that in repo S, which has patch P0 and patch P2 but not yet patch P1, line 99 happens to be "hello world", but it is a *different* hello world than the one which is changed by patch P1 over in repo S. Then a simple textual merge which takes into account only the current working states of S and R will *appear* clean. It will not insert any conflict markers, will not send output to the user warning him to look at this one more carefully, etc. But it will almost always be semantically wrong. Likewise, an example that I learned from reading an Arch document, suppose there is a ./somedir/Makefile in R, and in repo S it gets mv'ed into ./somedir/somesubdir/Makefile and a *new* file ./somedir/Makefile gets created. Meanwhile, in R, the contents of ./somedir/Makefile gets changed. Please observe that darcs patch commutation does the right thing here, where a more naive algorithm would apply the change to the wrong file. I'm aware of the issue of fileIds, but I prefer to contrast two algorithms here: no-file-Id-simple-textual-merge, and no-file-Id-darcs-commutation-merge. In this simple of example of "The Moved Makefile" it is clear that the naive algorithm is more dangerous than the commutation algorithm. Geez, I hope somebody reads this. :-) zooko: here's a real world example that actually happened. I created a patch that made 'bzr merge' check whether there were local modifications, and abort if there were. I also created a pull command that used the merge function to apply changes to the local tree. I submitted these as separate patches. In 'pull', the branch revision was modified before merge was invoked. This made the tree look like it had local modifications. Since the default was to check for local modifications, pull was now broken. zooko: I agree with the core of what you're saying. I don't believe that there is any merge algorithm which can be free of semantic conflicts. Well, you could elevate every change merged in to the user as a conflict, but I wouldn't really call that a "merge algorithm". I mean, that's basically ediff/tkdiff. Not that it's a bad way to do things. I am paranoid, and habitually read diffs before committing, even when I haven't done a merge. The possibility of semantic conflicts from parallel edits is one of the main themes in the comparison to CVS that the original Vesta developers wrote long ago: http://www.vestasys.org/doc/comparison.html#CVS However, it's clear that you need to allow for parallel edits. Any time you do that, there must be some process of merging them. No matter how automated or manual that process is, semantic conflicts can go un-noticed. Also, zooko, there are degrees of merging. I always run the smoke test of compiling. The baz team runs the test suite for each merge. Not everyone just merges and commits. My personal belief is that the best way to deal with them is: 1. Always read the diff before comitting (though not all users will do this), 2. Read the diffs other people commit (i.e. peer review), 3. Have an automated test suite (though it's often difficult to achieve good coverage). Sorry, I guess that should also include: 0. Always build before committing, and never commit something that doesn't build. xorian: I just realized that the ability to uncommit might be handy. Commit, have that trigger the test suite, if the test suite fails, uncommit. xorian: Sensible advice. abentley: I could go on at length about automating the integration of parallel changes made by multiple users (we do this at Intel with hundresd of peopl constantly working), but instead I will simply mention Fowler's Continuous integration: http://www.martinfowler.com/articles/continuousIntegration.html ...and the free implementation of the idea: http://cruisecontrol.sourceforge.net/ zooko: Anyhow, if you now understand why patch commutation is a heuristic, that's great. Of course our automated integrations of parallel changes (on microprocessor projects at Intel) don't involve merging. Users are always invovled in that, and sanity-checking the result. It only automatically integrates parallel changes to different files (actually packages, but that's Vesta-ese). But of course there can be semantic conflicts that cross file boundaries, which is exactly what that tests for. I also don't understand the Makefile example. Wouldn't the wrong makefile have the wrong contents too, and therefore fail on a textual basis? abentley: Any time you take changes made in parallel and combine them, it's a heuristic and can get semantic conflicts without syntactic conflicts. A good merge algorithm (and I would put Darcs' patch commutation in that category) simply reduces the number of syntactic conflicts which the user has to be involved in resolving. xorian: agreed. If anyone would like to suggest improvements to the description of criss-cross merge here, I would appreciate it: http://wiki.vestasys.org/MergingFuture/Food4Thought/TrickyCases I've been trying to document some of what I've learned over the last 6 months (largely from this channel in one way or another) for people who haven't thought about this at all to help in the formulation of future development plans for Vesta. This particular subject came up at work a couple days ago, so that prodded me to get around to writing it up. Basically though this is just an expansion on the post Bram made to the git list with full text for the states in that example to make it easier to see. (I went through the example myself a while back, and it really helped me understand it.) 'history-aware merging' seems too vague to me. That could describe the process of selecting a BASE using history data. abently: A valid criticism. I don't recall where I got that phrase from. I don't suppose you'd care to suggest an alternative? is going AFK for a while now, will read any comments when he returns You might also be interested to read David Allouche's description: http://lists.gnu.org/archive/html/gnu-arch-users/2004-09/msg00279.html I dunno, it's something like 'block-identity' merging. is reading abentley real-world example. abentley: if I understand correctly, your real-world example is a case where there were two changes, syntactically separate (so that syntactic detection of dependency won't work), but semantically related, so that applying both of the changes together resulted in a broken system. I'm not going to write another monologue, so I'll just wait patiently and see if abentley wants to confirm my understanding or else further enlightenme. :-) thanks zooko for the monologue :) zooko: correct. You said: However, I find it hard to think of more "realistic" examples in the realm of software development. So this was a more realistic example. As in real :-) Okay, I've understood your example.