Merge pull request #1084 from tesshuflower/restic_v0.16.4

Update restic to v0.16.4

CHANGELOG.md

@@ -10,7 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 
 - Syncthing upgraded to v1.26.1
-- Restic upgraded to v0.16.3
+- Restic upgraded to v0.16.4
 
 ## [0.8.0]
 

mover-restic/SOURCE_VERSIONS

@@ -1,2 +1,2 @@
-https://github.com/restic/restic.git v0.16.3 abca1124041f9c83227d253142dcc713e7b3914c
+https://github.com/restic/restic.git v0.16.4 3786536dc18ef27aedcfa8e4c6953b48353eee79
 https://github.com/minio/minio-go.git v7.0.66 5415e6c72a71610108fe05ee747ac760dd40094f

mover-restic/restic/CHANGELOG.md

@@ -1,5 +1,6 @@
 # Table of Contents
 
+* [Changelog for 0.16.4](#changelog-for-restic-0164-2024-02-04)
 * [Changelog for 0.16.3](#changelog-for-restic-0163-2024-01-14)
 * [Changelog for 0.16.2](#changelog-for-restic-0162-2023-10-29)
 * [Changelog for 0.16.1](#changelog-for-restic-0161-2023-10-24)
@@ -32,6 +33,57 @@
 * [Changelog for 0.6.0](#changelog-for-restic-060-2017-05-29)
 
 
+# Changelog for restic 0.16.4 (2024-02-04)
+The following sections list the changes in restic 0.16.4 relevant to
+restic users. The changes are ordered by importance.
+
+## Summary
+
+ * Fix #4677: Downgrade zstd library to fix rare data corruption at max. compression
+ * Enh #4529: Add extra verification of data integrity before upload
+
+## Details
+
+ * Bugfix #4677: Downgrade zstd library to fix rare data corruption at max. compression
+
+   In restic 0.16.3, backups where the compression level was set to `max` (using
+   `--compression max`) could in rare and very specific circumstances result in
+   data corruption due to a bug in the library used for compressing data. Restic
+   0.16.1 and 0.16.2 were not affected.
+
+   Restic now uses the previous version of the library used to compress data, the
+   same version used by restic 0.16.2. Please note that the `auto` compression
+   level (which restic uses by default) was never affected, and even if you used
+   `max` compression, chances of being affected by this issue are small.
+
+   To check a repository for any corruption, run `restic check --read-data`. This
+   will download and verify the whole repository and can be used at any time to
+   completely verify the integrity of a repository. If the `check` command detects
+   anomalies, follow the suggested steps.
+
+   https://github.com/restic/restic/issues/4677
+   https://github.com/restic/restic/pull/4679
+
+ * Enhancement #4529: Add extra verification of data integrity before upload
+
+   Hardware issues, or a bug in restic or its dependencies, could previously cause
+   corruption in the files restic created and stored in the repository. Detecting
+   such corruption previously required explicitly running the `check --read-data`
+   or `check --read-data-subset` commands.
+
+   To further ensure data integrity, even in the case of hardware issues or
+   software bugs, restic now performs additional verification of the files about to
+   be uploaded to the repository.
+
+   These extra checks will increase CPU usage during backups. They can therefore,
+   if absolutely necessary, be disabled using the `--no-extra-verify` global
+   option. Please note that this should be combined with more active checking using
+   the previously mentioned check commands.
+
+   https://github.com/restic/restic/issues/4529
+   https://github.com/restic/restic/pull/4681
+
+
 # Changelog for restic 0.16.3 (2024-01-14)
 The following sections list the changes in restic 0.16.3 relevant to
 restic users. The changes are ordered by importance.

mover-restic/restic/VERSION

@@ -1 +1 @@
-0.16.3
+0.16.4

mover-restic/restic/changelog/0.16.4_2024-02-04/issue-4529

@@ -0,0 +1,18 @@
+Enhancement: Add extra verification of data integrity before upload
+
+Hardware issues, or a bug in restic or its dependencies, could previously cause
+corruption in the files restic created and stored in the repository. Detecting
+such corruption previously required explicitly running the `check --read-data`
+or `check --read-data-subset` commands.
+
+To further ensure data integrity, even in the case of hardware issues or
+software bugs, restic now performs additional verification of the files about
+to be uploaded to the repository.
+
+These extra checks will increase CPU usage during backups. They can therefore,
+if absolutely necessary, be disabled using the `--no-extra-verify` global
+option. Please note that this should be combined with more active checking
+using the previously mentioned check commands.
+
+https://github.com/restic/restic/issues/4529
+https://github.com/restic/restic/pull/4681

mover-restic/restic/changelog/0.16.4_2024-02-04/issue-4677

@@ -0,0 +1,19 @@
+Bugfix: Downgrade zstd library to fix rare data corruption at max. compression
+
+In restic 0.16.3, backups where the compression level was set to `max` (using
+`--compression max`) could in rare and very specific circumstances result in
+data corruption due to a bug in the library used for compressing data. Restic
+0.16.1 and 0.16.2 were not affected.
+
+Restic now uses the previous version of the library used to compress data, the
+same version used by restic 0.16.2. Please note that the `auto` compression
+level (which restic uses by default) was never affected, and even if you used
+`max` compression, chances of being affected by this issue are small.
+
+To check a repository for any corruption, run `restic check --read-data`. This
+will download and verify the whole repository and can be used at any time to
+completely verify the integrity of a repository. If the `check` command detects
+anomalies, follow the suggested steps.
+
+https://github.com/restic/restic/issues/4677
+https://github.com/restic/restic/pull/4679

mover-restic/restic/cmd/restic/global.go

@@ -43,7 +43,7 @@ import (
 	"golang.org/x/term"
 )
 
-var version = "0.16.3"
+var version = "0.16.4"
 
 // TimeFormat is the format used for all timestamps printed by restic.
 const TimeFormat = "2006-01-02 15:04:05"
@@ -67,6 +67,7 @@ type GlobalOptions struct {
 	CleanupCache    bool
 	Compression     repository.CompressionMode
 	PackSize        uint
+	NoExtraVerify   bool
 
 	backend.TransportOptions
 	limiter.Limits
@@ -139,6 +140,7 @@ func init() {
 	f.BoolVar(&globalOptions.InsecureTLS, "insecure-tls", false, "skip TLS certificate verification when connecting to the repository (insecure)")
 	f.BoolVar(&globalOptions.CleanupCache, "cleanup-cache", false, "auto remove old cache directories")
 	f.Var(&globalOptions.Compression, "compression", "compression mode (only available for repository format version 2), one of (auto|off|max) (default: $RESTIC_COMPRESSION)")
+	f.BoolVar(&globalOptions.NoExtraVerify, "no-extra-verify", false, "skip additional verification of data before upload (see documentation)")
 	f.IntVar(&globalOptions.Limits.UploadKb, "limit-upload", 0, "limits uploads to a maximum `rate` in KiB/s. (default: unlimited)")
 	f.IntVar(&globalOptions.Limits.DownloadKb, "limit-download", 0, "limits downloads to a maximum `rate` in KiB/s. (default: unlimited)")
 	f.UintVar(&globalOptions.PackSize, "pack-size", 0, "set target pack `size` in MiB, created pack files may be larger (default: $RESTIC_PACK_SIZE)")
@@ -453,8 +455,9 @@ func OpenRepository(ctx context.Context, opts GlobalOptions) (*repository.Reposi
 	}
 
 	s, err := repository.New(be, repository.Options{
-		Compression: opts.Compression,
-		PackSize:    opts.PackSize * 1024 * 1024,
+		Compression:   opts.Compression,
+		PackSize:      opts.PackSize * 1024 * 1024,
+		NoExtraVerify: opts.NoExtraVerify,
 	})
 	if err != nil {
 		return nil, errors.Fatal(err.Error())

mover-restic/restic/doc/047_tuning_backup_parameters.rst

@@ -60,6 +60,20 @@ only applied for the single run of restic. The option can also be set via the en
 variable ``RESTIC_COMPRESSION``.
 
 
+Data Verification
+=================
+
+To prevent the upload of corrupted data to the repository, which can happen due
+to hardware issues or software bugs, restic verifies that generated files can
+be decoded and contain the correct data beforehand. This increases the CPU usage
+during backups. If necessary, you can disable this verification using the
+``--no-extra-verify`` option of the ``backup`` command. However, in this case
+you should verify the repository integrity more actively using
+``restic check --read-data`` (or the similar ``--read-data-subset`` option).
+Otherwise, data corruption due to hardware issues or software bugs might go
+unnoticed.
+
+
 File Read Concurrency
 =====================