Epic Games announces Lore version control system

[1] https://github.com/debarshibasak/assets

In fairness, p4 predates most of the VCSes we consider "modern", so I empathize with a lot of the underlying architecture decisions. However, it has and continues to utterly fail at improving at a reasonable pace.

For example:

  - p4 tracks file metadata of client workspaces on the server (sync'ed locally, opened for edit, file revision, etc) and uses this as the basis to avoid doing unneeded work.  If this becomes desync'ed, a reconcile or force sync must be used.  A reconcile can take hours, potentially days; it tries do detect file moves by default, so likely at least O(c^n) for some c>1. I have never personally seen a default reconcile operation _complete_ over any modestly large game code base, and in practice, people accumulate a litany of workarounds and scripts to fix this for themselves.

  - Scripting p4 is a nightmare. Documentation is poor, schemas do not exist, and all the language-specific libraries are just thin wrappers over its C++ API.

  - By default, p4 "helps" you with text files by "correcting" line endings on sync or even converting between encodings. This works until you have a mixed-OS environment, and discover a part of the pipechain that _must_ have a certain style.  There are various levers to pull to make this better, but I've yet to find something fool proof.

  - By default, p4 keeps flies read-only, only unlocking them when explicitly marked as being edited.  This means, to avoid having to do this manually, every tool you use needs to be p4-aware.  Or, you can turn this off, and choose to contend reconcile instead. (See above)

  - Branching a modest game project, with, say, Unreal source code, can take hours.  And this is the quick version where you ask the server to simply create new metadata, with no file transfer to a client.

  - p4 is licensed by the user-account. Every user entity in p4 not intended exclusively for performing backups and maintenance operations counts toward this, including users required to integrate with other services.  Plus, often times, these integration users must have admin access to be useful. The security posture is horrific.

If perforce is the best there is out there for large binary asset management, then there is a blue ocean worth of potential improvement for git.

Perforce is a piece of crap, a relic of the 20th century that must die in a fiery inferno.

The objects are referenced by trees. A tree is just a directory.

The trees are then referenced by commits and/or tags into a DAG with named pointers into various parts of it (which are your branch and tag references):

https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Because it would be terribly in-efficient to have a bunch of loose objects, git periodically groups them together into packs. To save space, the objects are compressed against one another (delta compression) within the packs.

https://git-scm.com/docs/git-pack-objects

https://github.com/git/git/blob/master/Documentation/technic...

When pushing or pulling, the git transfer protocol basically enumerates what objects each side has so that it only needs to transfer the difference. On top of that, it delta compresses the objects on each side that aren't already grouped into packs against each other to save space.

https://github.com/git/git/blob/master/Documentation/technic...

Because git is an open-source project written by nerds, it shows you all of this information. Feel free to ignore it!

But if you really want to know, it's all documented both in the git book and git documentation directory, both linked above.

(Caveat: I'm working from memory and surely got some detail at least slightly wrong.)

The git support for new employees drops basically to zero.

I’d be a bit worried if git didn’t heave that particular contented sigh when I ask it to push

Not saying Lore's approach is bad, but sometimes "worse is better".

Don't be too hard on Lore.

I'm not just picking nits here. And this is not cynicism.

so there you go.

That already warrants different storage architectures.

> Open-source the desktop client so the community can build on its full graphical experience, not just download it. An early desktop client already exists as a binary download, but it isn't open source yet — it depends on some proprietary components, including Epic's internal design system. We're working to make all of it available in the open so that the client can ship as source alongside the rest of Lore. Lore is an open project, so it is important that the desktop client — which will be one of the main ways many people will interact with Lore — is also fully open so that the community is free to review, extend, and shape it.

they do say they will open source it, but who knows:

"It isn’t open source yet—it currently depends on some proprietary components, including Epic’s internal design system—but we’re committed to open-sourcing it in the future"

EDIT: and then an example for the merge stuff I couldn't find while typing before: https://mergiraf.org/ and HN discussion a few years ago: https://news.ycombinator.com/item?id=42093756

Lore is meant for situations where your repository is going to contain gigabytes of binary files, such as art assets for games. Git is technically great at everything but that, and even the external solutions for that situation still kind of suck.

Anyway it's probably fine software and I am genuinely going to give it a shot for a usecase I have involving large image files. But the LLM-generated docs don't inspire confidence.

[1] https://epicgames.github.io/lore/explanation/system-design/

[2] They literally have a section header "10.1 Revision state as a 320-byte fragment". The byte size isn't even relevant in the code as an implementation detail, much less belongs in a design doc. No one read this doc before publishing it.

On the flip side, I expect the project itself will be workable - well, assuming they're actually using it themselves! UE is a big pile of Stuff on its own, and Fortnite must have god knows how much additional crap in there, so if this is (or will be) their replacement for Perforce internally, then it'll be getting a good deal of testing. (If they're just chucking it over the wall, though... well, sheesh, you first...)

(Perforce is the standard thing for games, and pretty well it works too, and hopefully this will deliver it a well-needed kick. It was sold to private equity about 10 years ago, and it feels like they've been coasting ever since. (Perhaps users in other sectors are happier though?))

> 3.2 Explicit non-goals¶

> Peer-to-peer decentralization. Lore is centralized by design. Two clients communicate through the remote, not directly.