Add destination concurrency support to znapzendzetup

[nickilby]

Make Destination Concurrency Opt-In with Backward Compatibility

Executive Summary

This pull request introduces an improvement for znapzend users by making destination replication concurrency opt-in. The change preserves 100% backward compatibility with existing configurations while introducing safer defaults for new backup sets.

Key Impact:

• New backup sets default to serial (one-at-a-time) destination processing, requiring explicit opt-in for parallelism
• Existing configurations are unaffected and maintain their current behavior automatically
• Users can enable parallelism with --dst-concurrency flag during create/edit operations
• Load control is now an explicit operator decision, reducing surprise resource saturation

---

Problem Statement

Current Behavior (Before)

When a znapzend backup plan is created with multiple destination targets, znapzend processes destinations serially by default (one destination at a time):

SRC dataset
  ├─→ DST:a (waits)
  ├─→ DST:b (waits)
  └─→ DST:c (waits)

Current serial processing is safe and predictable, but it has limitations:

• Slow backup completion: Each destination must fully complete before the next begins
• Long waiting time: On systems with many destinations, total backup window becomes unacceptably long

Why This Is A Need For Improvement

1. Sequential bottleneck: For operators with 5+ destinations, backup windows stretch to hours or days
2. No parallelism option: Operators needing faster backup completion have no way to speed up destination processing
3. No resource control: There's no mechanism to enable faster backups with bounded resource usage
4. Capacity underutilization: Network and disk I/O sit idle while waiting for serial destination transfers to complete

Real-World Impact Example

An operator configures 3 remote backup destinations for their production database:

znapzendzetup create \
  SRC '1d=>12h,3d=>1d' tank/data \
  DST:hq 'plan...' hq-backup:tank/data \
  DST:dr 'plan...' dr-backup:tank/data \
  DST:cloud 'plan...' cloud-backup:tank/data

Result: Destinations process serially, each taking 5 minutes. Total backup window: 15+ minutes per cycle. Network link remains at 20% utilization (could safely push to 60% with stricter concurrency limits).

---

Solution: Explicit Opt-In for Parallelism

This PR introduces an explicit concurrency control mechanism that allows operators to choose parallelism when beneficial, while maintaining safe serial defaults.

Key Design Principles

1. Zero breaking changes: Existing serial behavior continues unchanged
2. Explicit is better than implicit: Parallelism becomes an intentional, documented decision
3. Operator-controlled: Full control over concurrency levels and resource impact
4. Backward compatible: Old and new configs coexist peacefully
5. Progressive enhancement: Operators can enable parallelism only when capacity is available

---

Technical Implementation

New Property: dst_concurrency_enabled

A new ZFS property controls whether destination concurrency is active:

Value	Behavior
on	Destinations run in parallel (operator opted in)
off	Destinations run serially, one-at-a-time (default for new configs)

New CLI Flag: --dst-concurrency

The --dst-concurrency flag enables parallel destination processing with optional worker limit:

# Enable parallelism to ALL destinations at once (processes all concurrently)
znapzendzetup create --dst-concurrency ...

# Enable parallelism LIMITED to 2 concurrent workers (processes max 2 at a time)
znapzendzetup create --dst-concurrency=2 ...

# Disable flag (serial processing - the default)
znapzendzetup create ...  # No flag = serial by default

---

Behavior Matrix: All Scenarios

Scenario 1: New Backup Set (No Flag)

znapzendzetup create --recursive --donotask \
  SRC '1d=>12h,3d=>1d' tank/apps \
  DST:a '31d=>1d' backup-a:tank/apps \
  DST:b '31d=>1d' backup-b:tank/apps

Result:

• Property: dst_concurrency_enabled=off
• Behavior: Serial processing (DST:a completes, then DST:b) — same as current default
• Rationale: Maintains existing safe behavior; user must explicitly opt in for parallelism

---

Scenario 2: New Backup Set with Full Parallelism

znapzendzetup create --recursive --dst-concurrency --donotask \
  SRC '1d=>12h,3d=>1d' tank/apps \
  DST:a '31d=>1d' backup-a:tank/apps \
  DST:b '31d=>1d' backup-b:tank/apps \
  DST:c '14d=>1d,3m=>1w' backup-c:tank/apps

Result:

• Property: dst_concurrency_enabled=on
• Property: dst_concurrency (unset, defaults to all destinations)
• Behavior: Parallel processing (all 3 run simultaneously) — NEW capability
• Rationale: Operator explicitly enabled parallelism for faster backup completion

---

Scenario 3: New Backup Set with Bounded Parallelism

znapzendzetup create --recursive --dst-concurrency=2 --donotask \
  SRC '1d=>12h,3d=>1d' tank/apps \
  DST:a '31d=>1d' backup-a:tank/apps \
  DST:b '31d=>1d' backup-b:tank/apps \
  DST:c '14d=>1d,3m=>1w' backup-c:tank/apps

Result:

• Property: dst_concurrency_enabled=on
• Property: dst_concurrency=2
• Behavior: Parallel processing, max 2 workers (e.g., DST:a + DST:b run together, wait, then DST:c) — NEW capability
• Rationale: Operator controls parallelism with resource limits

---

Scenario 4: Existing Configuration (No Changes Needed)

Configuration created before this PR:
  dst_concurrency_enabled: (property does not exist)
  dst_concurrency: (may or may not be set)

Result:

• Behavior: Exactly as before — unchanged for that backup set
• No action required

---

Scenario 5: Editing Existing Config (Without Concurrency Flag)

znapzendzetup edit --donotask --mbuffersize=8G \
  SRC tank/apps \
  DST:a backup-a:tank/apps

Result:

• Property: dst_concurrency_enabled (unchanged)
• Behavior: No change to concurrency (only mbuffersize updated)
• Rationale: Edits don't surprise users with new behavior

---

Scenario 6: Editing Existing Config (With Concurrency Flag)

znapzendzetup edit --donotask --dst-concurrency=3 \
  SRC tank/apps \
  DST:a backup-a:tank/apps \
  DST:b backup-b:tank/apps \
  DST:c backup-c:tank/apps

Result:

• Property: dst_concurrency_enabled=on
• Property: dst_concurrency=3
• Behavior: Parallelism enabled with limit of 3 — operator opt-in
• Rationale: Explicit operator change to parallel mode

---

Configuration Examples

Example 1: Conservative Setup (Default for New Configs)

# Create without concurrency flag
znapzendzetup create --recursive --donotask \
  --mbuffersize=4G \
  SRC '1d=>12h,3d=>1d' tank/home \
  DST:a '31d=>1d' backup-server-1:backup/home \
  DST:b '31d=>1d' backup-server-2:backup/home

Config stored in ZFS:

org.znapzend:enabled=on
org.znapzend:recursive=on
org.znapzend:src_plan=1d=>12h,3d=>1d
org.znapzend:dst_concurrency_enabled=off
org.znapzend:dst_a=backup-server-1:backup/home
org.znapzend:dst_a_plan=31d=>1d
org.znapzend:dst_b=backup-server-2:backup/home
org.znapzend:dst_b_plan=31d=>1d

Runtime behavior:

Cycle 1:
  [14:00] Start sending to DST:a
  [14:05] DST:a complete
  [14:05] Start sending to DST:b
  [14:10] DST:b complete
Total time: ~10 minutes, constant moderate load (same as current)

---

Example 2: High-Throughput Setup (Opt-In Parallelism)

# Create WITH concurrency flag and limit to enable parallelism
znapzendzetup create --recursive --dst-concurrency=2 --donotask \
  --mbuffersize=4G \
  SRC '1d=>12h,3d=>1d' tank/data \
  DST:hq '31d=>1d' hq-backup:tank/data \
  DST:dr '31d=>1d' dr-backup:tank/data \
  DST:cloud '14d=>1d,3m=>1w' cloud-backup:tank/data

Config stored in ZFS:

org.znapzend:enabled=on
org.znapzend:recursive=on
org.znapzend:src_plan=1d=>12h,3d=>1d
org.znapzend:dst_concurrency_enabled=on
org.znapzend:dst_concurrency=2
org.znapzend:dst_hq=hq-backup:tank/data
org.znapzend:dst_hq_plan=31d=>1d
org.znapzend:dst_dr=dr-backup:tank/data
org.znapzend:dst_dr_plan=31d=>1d
org.znapzend:dst_cloud=cloud-backup:tank/data
org.znapzend:dst_cloud_plan=14d=>1d,3m=>1w

Runtime behavior:

Cycle 1:
  [14:00] Start sending to DST:hq and DST:dr (parallel, 2 workers)
  [14:05] Both complete
  [14:05] Start sending to DST:cloud (1 worker, would be 3 if more available)
  [14:08] DST:cloud complete
Total time: ~8 minutes (faster!), 2x peak load but bounded

---

Example 3: Upgrading Existing Config (No Changes)

An operator has an existing config from before this PR:

org.znapzend:enabled=on
org.znapzend:src_plan=1d=>12h,3d=>1d
org.znapzend:dst_a=backup-a:tank/data
org.znapzend:dst_a_plan=31d=>1d
org.znapzend:dst_b=backup-b:tank/data
org.znapzend:dst_b_plan=31d=>1d
(Note: No dst_concurrency_enabled property, no dst_concurrency property)

After upgrade to this PR version:

• No action required
• Property dst_concurrency_enabled is not present
• Runtime follows the marker-not-set path, preserving prior behavior for that backup set
• Behavior is identical to before the upgrade—completely transparent upgrade

If operator NOW wants to enable parallelism:

znapzendzetup edit --donotask --dst-concurrency=2 \
  SRC tank/data \
  DST:a backup-a:tank/data \
  DST:b backup-b:tank/data

Result: dst_concurrency_enabled=on and dst_concurrency=2 are added to config, enabling 2-worker parallelism going forward.

---

Changed Behavior Summary

For New Configurations

Scenario	Before PR	After PR
No concurrency flag	Serial (default)	Serial (default) — unchanged
--dst-concurrency	Must manually set concurrency in config	All parallel (explicit opt-in) — NEW
--dst-concurrency=N	Must manually set limit in config	Limited to N parallel (explicit opt-in) — NEW

For Existing Configurations

Scenario	Before PR	After PR
Already works	Serial by default	Serial by default (unchanged)
No migration needed	N/A	Yes, zero migration effort

---

Migration and Upgrade Guide

For System Administrators

TL;DR: Do nothing. Existing systems are unaffected.

1. Upgrade znapzend to this version
2. No configuration changes required
3. All existing backup sets continue working as before
4. New schedules default to serial (if you want parallelism, use --dst-concurrency)

For New Deployments

Start with the safe default and opt-in to parallelism only if needed:

# Safe default: use this for most cases
znapzendzetup create --recursive SRC ... DST ...

# Only if you need high throughput and understand the load impact
znapzendzetup create --recursive --dst-concurrency=2 SRC ... DST ...

Decision Tree: Should I Use --dst-concurrency?

Do you have multiple destinations?
  YES → Do you have spare I/O capacity?
    YES → Do you need fast backup completion?
      YES → Use --dst-concurrency (with optional =N limit)
      NO → Don't use the flag (serial is fine)
    NO → Don't use the flag (serial mode protects your workloads)
  NO → Not applicable (single destination is always serial)

---

Implementation Details

Files Changed

1. bin/znapzendzetup

   • CLI option parsing: Added new --dst-concurrency:s flag (optional argument supporting bare flag or value)
   • Create mode: Detect flag presence, set dst_concurrency_enabled=off by default
   • Edit mode: Only update concurrency if flag is explicitly passed
   • Help text: Added documentation of new option

2. lib/ZnapZend.pm

   • Runtime resolution logic: Check dst_concurrency_enabled first, then use the marker-not-set compatibility path
   • Concurrency calculation:
     • If enabled=off → 1 worker (serial)
     • If enabled=on + no limit → all destinations
     • If enabled=on + limit → respect limit
     • If enabled unset → marker-not-set compatibility path

3. lib/ZnapZend/Config.pm

   • Validation: Added check for dst_concurrency_enabled values (on|off)
   • Existing validation: dst_concurrency must be integer >= 1 (unchanged)

4. doc/znapzendzetup.pod

   • Help documentation: Updated with new option syntax and behavior explanation

5. README.md

   • User guide: Added examples of parallelism configuration
   • Notes: Explained new default behavior and opt-in model

6. t/znapzendzetup.t

   • Tests: Added coverage for bare --dst-concurrency flag
   • Existing tests: Retained for regression coverage

---

Safety and Risk Mitigation

Safety Measures

1. Compatibility path preserved: Existing configs continue unchanged
2. Explicit is default: New configs start with safest option (serial)
3. Clamping: Concurrency limited to destination count (no worker count > destinations)
4. Warnings: High concurrency (>8 workers) triggers warning log
5. Event loop protection: Falls back to serial if already in async context

Validation

• dst_concurrency_enabled must be on or off (dies on invalid value)
• dst_concurrency must be positive integer >= 1 (existing rule, unchanged)
• Both properties are optional (allowing pre-marker configs)

Testing

• Unit tests: Create/edit with/without flag
• Config validation tests: Property value bounds
• Regression tests: Existing behavior unchanged
• Note: Full runtime tests require Perl environment (not run in this environment)

---

Logging and Observability

Runtime Decision Logging

When processing backup sets, znapzend emits per-destination send logs. Sanitized example:

[2026-03-27 12:05:01.01806] [123456] [debug] sending snapshots from srcpool/appdata to site-a:backuppool/appdata
[2026-03-27 12:05:01.02233] [123457] [debug] sending snapshots from srcpool/appdata to site-b:backuppool/appdata
[2026-03-27 12:05:01.02336] [123458] [debug] sending snapshots from srcpool/appdata to site-c:backuppool/appdata

When destination concurrency is enabled, multiple destinations start at nearly the same
timestamp (often with different worker PIDs), indicating parallel send workers.

---

Backward Compatibility Promise

This PR Guarantees

✅ Zero breaking changes for systems using existing backup sets
✅ Automatic compatibility-path detection — no manual migration required
✅ Composable with future enhancements (other properties are unaffected)
✅ Invertible — can be undone with explicit property edits if needed

This PR Does NOT

❌ Change behavior of existing backup sets

---

Operational Safety

• No existing deployments are affected
• Users get clear logging of which mode is active
• Warnings for high concurrency values
• No silent behavior changes on existing configs

---

Questions and Answers

Q: Will my existing backups break?

A: No. Existing backup sets continue working exactly as before. This PR does not silently change behavior on existing sets. The property dst_concurrency_enabled is optional and parallelism remains an explicit operator choice.

Q: How do I enable parallelism in new configs?

A: Use --dst-concurrency during create: znapzendzetup create --dst-concurrency .... Use --dst-concurrency=2 to limit to 2 parallel workers. Without the flag, new configs default to serial processing for safety.

Q: What if I want to edit an existing config without changing concurrency?

A: Just don't use the --dst-concurrency flag during edit. Only properties you explicitly pass are updated (and concurrency behavior remains unchanged).

Q: Can I convert an existing config to use the new opt-in concurrency?

A: Yes. Run znapzendzetup edit --dst-concurrency=<limit> ... and it will add the marker and enable parallelism with the specified limit.

Q: What happens if dst_concurrency_enabled is on but the numeric limit is not set?

A: It defaults to "all destinations in parallel" (unbounded parallelism for maximum throughput).

Q: Why use a separate marker property instead of checking if dst_concurrency exists?

A: The marker (dst_concurrency_enabled) cleanly separates "user wants parallelism" (on/off) from "how many workers" (the numeric limit). This avoids overloading one property for two meanings and keeps concurrency intent explicit.

---

Conclusion

This PR adds explicit opt-in parallelism for destination processing while maintaining 100% backward compatibility with existing deployments. The feature transparently enhances the znapzend toolbox for operators who need faster backup completion, while keeping safe serial defaults for new configurations.

The implementation is minimal and focused: a single marker property (dst_concurrency_enabled) and a CLI flag (--dst-concurrency) enable parallelism when desired. Users upgrading to this version need take no action; their existing backup schedules continue working unchanged. New deployments benefit from the safer default and have the option to opt-in to parallelism when capacity permits.

[oetiker]

thanks, please update the CHANGES file

[nickilby]

Hi CHANGES file updated, and tests corrected

[oetiker]

you set it to draft ?