git-annex 10.20241202

git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, checksumming time, or disk space. Even without file content tracking, being able to manage files with git, move files around and delete files with versioned directory trees, and use branches and distributed clones, are all very handy reasons to use git. And annexed files can co-exist in the same git repository with regularly versioned files, which is convenient for maintaining documents, Makefiles, etc that are associated with annexed files but that benefit from full revision control. When a file is annexed, its content is moved into a key-value store, and a symlink is made that points to the content. These symlinks are checked into git and versioned like regular files. You can move them around, delete them, and so on. Pushing to another git repository will make git-annex there aware of the annexed file, and it can be used to retrieve its content from the key-value store.

Tags git archive haskell
License Mixed
State stable

Recent Releases

10.2024120205 Dec 2024 17:05 minor bugfix: Add: Consistently treat files in a dotdir as dotfiles, even. When ran inside that dotdir. Add: When adding a dotfile as a non-large file, mention that it's a. Dotfile. P2phttp: Added --directory option which serves multiple git-annex. Repositories located inside a directory. When remote.name.annexUrl is an annex+http(s) url, that. Uses the same hostname as remote.name.url, which is itself a http(s) Url, they are assumed to share a username and password. This avoids Unnecessary duplicate password prompts. Git-remote-annex: a reversion introduced in version 10.20241031. That broke cloning from a special remote. Git-remote-annex: cloning from a special remote on a crippled. Filesystem. Git-remote-annex: gy behavior when annex.stalldetection is. Configured. Git-remote-annex: Require git version 2.31 or newer, since old. Ones had a gy git bundle command. S3: Support versioning=yes with a readonly bucket. (Needs aws-0.24.3). S3: Send git-annex or other configured User-Agent. (Needs aws-0.24.3). S3: infinite loop and memory blowup when importing from an. Unversioned S3 bucket that is large enough to need pagination. S3: Use significantly less memory when importing from a. Versioned S3 bucket. Vpop: Only update state after successful checkout. -- Joey Hess Mon, 02 Dec 2024 12:31:08 -0400.
10.2024103128 Nov 2024 14:05 minor bugfix: Sped up proxied downloads from special remotes, by streaming. Added GETORDERED request to external special remote protocol. When the external special remote responds with ORDERED, it can stream. Through a proxy. P2phttp: Support serving unauthenticated users while requesting. Authentication for operations that need it. Eg, --unauth-readonly Can be combined with --authenv. P2phttp: Allow unauthenticated users to lock content by default. P2phttp: Added --unauth-nolocking option to prevent unauthenticated. Users from locking content. Allow enabling the servant build flag with older versions of stm. Allowing building with ghc 9.0.2. Git-remote-annex: that prevented using it with external special. Remotes, leading to protocol error messages involving "GITMANIFEST". Adjust: Allow any order of options when combining --hide-missing with. Options like --unlock. Support P2P protocol version 4. This allows DATA-PRESENT to be sent. After PUT (and in the HTTP P2P protocol, v4/put has a data-present Parameter). When used with a proxy to a special remote like a S3 Bucket, this allows a custom client to upload content to S3 itself, And then use the P2P protocol to inform the proxy that the content has Been stored there, which will result in the same git-annex branch state Updates as sending DATA via the proxy. Hang when receiving a large file into a proxied special remote. -- Joey Hess Thu, 31 Oct 2024 17:19:56 -0400.
10.2024092725 Nov 2024 09:05 minor bugfix: Detect when a preferred content expression contains "not present". Which would lead to repeatedly getting and then dropping files, And make it never match. This also applies to "not balanced" and "not sizebalanced". --explain display of onlyingroup preferred content expression. Allow maxsize to be set to 0 to stop checking maxsize for a repository. That prevented anything being stored in an empty. Repository whose preferred content expression uses sizebalanced. Sim: New command, can be used to simulate networks of repositories. And see how preferred content and other configuration makes file Content flow through it. -- Joey Hess Wed, 25 Sep 2024 14:43:26 -0400.
10.2024083122 Nov 2024 20:25 minor feature: Special remotes configured with exporttree=yes annexobjects=yes. Can store objects in.git/annex/objects, as well as an exported tree. Support proxying to special remotes configured with. Exporttree=yes annexobjects=yes, and allow such remotes to be used as Cluster nodes. Post-retrieve: When proxying is enabled for an exporttree=yes. Special remote (or it is a cluster node) and the configured Remote.name.annex-tracking-branch is received, the tree is Exported to the special remote. Support "balanced=", "fullybalanced=", "sizebalanced=" and. "fullysizebalanced=" in preferred content expressions. Added --rebalance option. Added the annex.fullybalancedthreshhold git config. Maxsize: New command to tell git-annex how large the expected maximum. Size of a repository is, and to display repository sizes. Vicfg: Include maxsize configuration. Info: Improved speed by using new repository size tracking. Lookupkey: Allow using --ref in a bare repository. Export: Added --from option. Git-remote-annex: Store objects in exportree=yes special remotes in the same paths used by annexobjects=yes. This is a backwards. Compatible change. Updateproxy, updatecluster: Prevent using an exporttree=yes special. Remote that does not have annexobjects=yes, since it will not work. The config versioning=true is now reserved for use by versioned special. Remotes. External special remotes should not use that config for their Own purposes. -- Joey Hess Sat, 31 Aug 2024 19:48:17 -0400.
10.2024080822 Sep 2024 14:25 minor bugfix: Remove deoutput (to stderr) accidentially included in last version. When getting from a P2P HTTP remote, prompt for credentials when required, instead of failing. When proxying an upload to a special remote, verify the hash. Avoid loading cluster log at startup. -- Joey Hess Thu, 08 Aug 2024 15:26:26 -0400.
10.2024073102 Sep 2024 21:05 minor feature: New HTTP API that is equivilant to the P2P protocol. New p2phttp command to serve the HTTP API. annex+http and annex+https urls can be configured for remote.name.annexUrl to use the HTTP API to communicate with a server. This supports writable repositories, as well as accessing clusters and proxied remotes over HTTP. When a http remote has annex.url set to an annex+http url in the git config file on the website, it will be copied into remote.name.annexUrl the first time git-annex uses the remote. assistant: a race condition that could cause a pointer file to get ingested into the annex. Avoid potential data loss in unlikely situations where git-annex-shell or git-annex remotedaemon is killed while locking a key to prevent its removal. When proxying a download from a special remote, avoid unncessary hashing. When proxying an upload to a special remote, verify the hash. Propagate --force to git-annex transferrer. Added a build flag for servant, enabling annex+http urls and git-annex p2phttp. Added a dependency on the haskell clock library. Updated stack.yaml to nightly-2024-07-29. -- Joey Hess Wed, 31 Jul 2024 14:02:21 -0400.
10.2024070103 Jul 2024 02:45 minor bugfix: Git-annex remotes can now act as proxies that provide access to. Their remotes. Configure this with remote.name.annex-proxy And the git-annex update proxy command. Clusters are now supported. These are collections of nodes that can be accessed as a single entity, accessed by one or more gateway. Repositories. Added git-annex initcluster, updatecluster, and extendcluster commands. a where interrupting git-annex while it is updating the. Git-annex branch for an export could later lead to git fsck Complaining about missing tree objects. Tab completion of options like --from now includes special remotes, as well as proxied remotes and clusters. Tab completion of many commands like info and trust now includes. Remotes. P2P protocol version 2. Windows build with Win32 2.13.4+ Thanks, Oleg Tolmatcev. When --defilter or annex.defilter is set, avoid propigating. Deoutput from git-annex-shell, since it cannot be filtered. -- Joey Hess Mon, 01 Jul 2024 15:11:48 -0400.
10.2024053131 May 2024 18:05 minor feature: Git-remote-annex: New program which allows pushing a git repo to a. Git-annex special remote, and cloning from a special remote. (Based on Michael Hanke's git-remote-datalad-annex.). Initremote, enableremote: Added --with-url to enable using. Git-remote-annex. When building an adjusted unlocked branch, make pointer files. Executable when the annex object file is executable. Group: Added --list option. Fsck: recent reversion that made it say it was checksumming files. Whose content is not present. Avoid the --fast option preventing checksumming in some cases it. Was not supposed to. Testremote: Really fsck downloaded objects. Typo. Thanks, Yaroslav Halchenko. -- Joey Hess Thu, 30 May 2024 17:53:13 -0400.
10.2024043002 May 2024 06:05 minor feature: : While redundant concurrent transfers were already. Prevented in most cases, it failed to prevent the case where Two different repositories were sending the same content to The same repository. Addurl, importfeed: Added --verifiable option, which improves. The safety of --fast or --relaxed by letting the content of Annexed files be verified with a checksum that is calculated on a later download from the web. This will become the default later. Added rclone special remote, which can be used without needing to install the git-annex-remote-rclone program. This needs a forthcoming version of rclone (1.67.0), which supports. "rclone gitannex". Sync, assist, import: Allow -m option to be specified multiple. Times, to provide additional paragraphs for the commit message. Reregisterurl: New command that can change an url from being. Used by a special remote to being used by the web remote. Annex.maxextensions configuration controls how many filename. Extensions to preserve. Find: --help for --copies. Thanks, Gergely Risko. Windows: escaping output to terminal when using old. Versions of MinTTY. Added dependency on unbounded-delays. -- Joey Hess Tue, 30 Apr 2024 15:26:32 -0400.
10.2024022728 Feb 2024 10:25 minor feature: Importfeed: Added --scrape option, which uses yt-dlp to screen scrape. The equivilant of an RSS feed. Importfeed --force: Don't treat it as a failure when an already. Downloaded file exists. (a behavior change introduced in 10.20230626.). Importfeed --force: Avoid creating duplicates of existing. Already downloaded files when yt-dlp or a special remote was used. Addurl, importfeed: Added --raw-except option. Stack.yaml: Update to lts-22.9 and use crypton. Assistant, undo: When committing, let the usual git commit. Hooks run. Added annex.commitmessage-command config. Pre-commit: Avoid committing the git-annex branch. (except when a commit is made in a view, which changes metadata). Pass --no-warnings to yt-dlp. -- Joey Hess Tue, 27 Feb 2024 12:58:30 -0400.
10.2024012931 Jan 2024 02:05 minor feature: Info: Added "annex sizes of repositories" table to the overall display. Import: Sped up import from special remotes. Import: Added --message/-m option. Support using commands that implement the Stateless OpenPGP command line. Interface, as an alternative to gpg. Currently only supported for encryption=shared special remotes. When annex.shared-sop-command is configured. Test: Test a specified Stateless OpenPGP command when. Run with eg --test-git-config annex.shared-sop-command=sqop Improve disk free space checking when transferring unsized keys to. Local git remotes. Added configs annex.stalldetection-download, annex.stalldetection-upload. Annex.bwlimit-download, annex.bwlimit-upload, And similar per-remote configs. Improve annex.stalldetection to handle remotes that update progress. Less frequently than the configured time period. External: Monitor file size when getting content from external. Special remotes and use that to update the progress meter, in case the external special remote program does not report progress. Added --expected-present file matching option. Webapp: Added --port option, and annex.port config. Assistant: When generating a gpg secret key, avoid hardcoding the. Key algorithm and size. -- Joey Hess Mon, 29 Jan 2024 13:12:00 -0400.
10.2023122710 Jan 2024 17:50 minor feature: * migrate: Support distributed migrations by recording each migration, and adding a --update option that updates the local repository incrementally, hard linking annex objects to their new keys. * pull, sync: When operating on content, automatically handle distributed migrations. * Added annex.syncmigrations config that can be set to false to prevent pull and sync from migrating object content. * migrate: Added --apply option that (re)applies all recorded distributed migrations to the objects in repository. * migrate: Support adding size to URL keys that were added with --relaxed, by running eg: git-annex migrate --backend=URL foo * When importing from a special remote, support preferred content expressions that use terms that match on keys (eg "present", "copies=1"). Such terms are ignored when importing, since the key is not known yet. Before, such expressions caused the import to fail. * Support git-annex copy/move --from-anywhere --to remote. * Make git-annex get/copy/move --from foo override configuration of remote.foo.annex-ignore, as documented. * Lower precision of timestamps in git-annex branch, which can reduce the size of the branch by up to 8 . * sync: Fix locking problems during merge when annex.pidlock is set. * Avoid a problem with temp file names ending in "." on certian filesystems that have problems with such filenames. * sync, push: Avoid trying to send individual files to special remotes configured with importtree=yes exporttree=no, which would always fail. * Fix a crash opening sqlite databases when run in a non-unicode locale. (Needs persistent-sqlite 2.13.3.)
7.2019100917 Oct 2019 17:39 minor feature: