backup: simple NAS backup plugin for KVM #9451

rohityadavcloud · 2024-07-26T06:42:22Z

This is a simple NAS backup plugin for KVM which may be later expanded for other hypervisors. This backup plugin aims to use shared NAS storage on KVM hosts such as NFS (or CephFS and others in future), which is used to backup fully cloned VMs for backup & restore operations. This may NOT be as efficient and performant as some of the other B&R providers, but maybe useful for some KVM environments who are okay to only have full-instance backups and limited functionality.

Design & Implementation follows the networker B&R plugin, which is simply:

Implement B&R plugin interfaces
Use cmd-answer pattern to execute backup and restore operations on KVM host when VM is running (or needs to be restored) - instead of a B&R API client, relies on answers from KVM agent which executes the operations
Backups are full VM domain snapshots, copied to a VM-specific folders on a NAS target (NFS) along with a domain XML
Backup uses libvirt feature: https://libvirt.org/kbase/live_full_disk_backup.html orchestrated via virsh/bash script (nasbackup.sh) as the libvirt-java lacks the bindings
Supported instance volume storage for restore operations: NFS & local storage

Refer the doc PR for feature limitations and usage details:
apache/cloudstack-documentation#429

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots

Each backup offering map to a Backup repository that takes in backup storage details (address/url, credentials/mount options):

Rest is same as B&R feature works; we assign such offering to a VM and backup once completed shows up in the list:

This is an experimental simple NAS backup plugin for KVM which may be later expanded for other hypervisors. This backup plugin aims to use shared NAS storage on KVM hosts such as NFS or CephFS, which is used to backup fully cloned VMs for backup & restore operations. This may not be as efficient and performant as some of the other B&R providers, but maybe useful for some KVM environments. Signed-off-by: Rohit Yadav <[email protected]>

codecov · 2024-07-26T06:53:16Z

Codecov Report

Attention: Patch coverage is 0.62696% with 951 lines in your changes missing coverage. Please review.

Project coverage is 15.61%. Comparing base (605534b) to head (9efdb57).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...rg/apache/cloudstack/backup/NASBackupProvider.java	0.00%	218 Missing ⚠️
...ce/wrapper/LibvirtRestoreBackupCommandWrapper.java	0.81%	121 Missing ⚠️
...apache/cloudstack/backup/RestoreBackupCommand.java	0.00%	66 Missing ⚠️
...udstack/api/response/BackupRepositoryResponse.java	0.00%	63 Missing ⚠️
...g/apache/cloudstack/backup/BackupRepositoryVO.java	0.00%	56 Missing ⚠️
...cloudstack/backup/BackupRepositoryServiceImpl.java	0.00%	46 Missing ⚠️
...rg/apache/cloudstack/backup/TakeBackupCommand.java	0.00%	44 Missing ⚠️
...rg/apache/cloudstack/backup/BackupManagerImpl.java	4.34%	44 Missing ⚠️
...user/backup/repository/AddBackupRepositoryCmd.java	0.00%	43 Missing ⚠️
.../apache/cloudstack/backup/DeleteBackupCommand.java	0.00%	34 Missing ⚠️
... and 18 more

Additional details and impacted files

@@             Coverage Diff              @@
##              main    #9451       +/-   ##
============================================
+ Coverage     4.12%   15.61%   +11.49%     
- Complexity       0    12118    +12118     
============================================
  Files          383     5555     +5172     
  Lines        31354   486283   +454929     
  Branches      5551    61315    +55764     
============================================
+ Hits          1293    75945    +74652     
- Misses       29915   401966   +372051     
- Partials       146     8372     +8226

Flag	Coverage Δ
uitests	`4.12% <ø> (+<0.01%)`	⬆️
unittests	`16.40% <0.62%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Rohit Yadav <[email protected]>

…red-storage

Signed-off-by: Rohit Yadav <[email protected]>

rohityadavcloud · 2024-08-01T12:46:40Z

Update: take backup & delete backup implemented for NFS; TBD/TODO more operations.

Each backup repository is for a specific zone & provider. Eack NAS bkcp offering is tied to a back repo & each bkup offer assinged to VM -> defined its backup repository Signed-off-by: Rohit Yadav <[email protected]>

Signed-off-by: Rohit Yadav <[email protected]>

rohityadavcloud · 2024-08-02T10:06:39Z

@blueorangutan package

blueorangutan · 2024-08-02T10:08:03Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

engine/schema/src/main/resources/META-INF/db/schema-41910to42000.sql

plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java

server/src/main/java/org/apache/cloudstack/backup/BackupManagerImpl.java

sureshanaparti

minor comments, overall code looks good.

…00.sql Co-authored-by: Suresh Kumar Anaparti <[email protected]>

Signed-off-by: Rohit Yadav <[email protected]> (cherry picked from commit ecfeb04026e1bbf13dc22bb0aa713677c7708bb8) Signed-off-by: Rohit Yadav <[email protected]>

…red-storage

Signed-off-by: Rohit Yadav <[email protected]>

rajujith

LGTM

Works with KVM EL8 ,EL9, Ubuntu22,Ubuntu24
Backup and restore works fine. All Full backups.
Supported backup repository NFS, tested using QNAP.
Parallel backups and restore
Parallel backups and restore per host.
Multiple Linux distros and Windows.
Multiple Backup repositories.
Backup schedules.
Admin and users can trigger backups.
Primary storages of type NFS and local storage.
Upgrade from 4.19.

Limitations observed and possible improvements( some are limitations of the B&R Framework itself)

Restoring expunged instances.
Restoring unmanaged instances.
More backup repository type support.
CKS support
Agent logging could be improved.

rohityadavcloud · 2024-09-05T12:58:28Z

@blueorangutan package

blueorangutan · 2024-09-05T13:00:03Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-09-05T14:19:13Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11006

Pearl1594 · 2024-09-05T14:48:35Z

Few comments will be addressed as improvements in a subsequent PR.

rohityadavcloud · 2024-09-05T15:02:05Z

@blueorangutan test

blueorangutan · 2024-09-05T15:04:03Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

rohityadavcloud · 2024-09-05T16:47:39Z

Merging this based on reviews, manual QA and GitHub Actions tests. Will continue further test matrix on health check PR.

blueorangutan · 2024-09-06T01:53:08Z

[SF] Trillian test result (tid-11369)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53228 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9451-t11369-kvm-ol8.zip
Smoke tests completed. 138 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_secure_vm_migration	`Error`	135.16	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	135.16	test_vm_life_cycle.py

blueorangutan · 2024-09-06T06:48:24Z

[SF] Trillian test result (tid-11377)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 54143 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9451-t11377-kvm-ol8.zip
Smoke tests completed. 139 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_mount_shared_fs	`Failure`	84.98	test_sharedfs_lifecycle.py
test_resize_shared_fs	`Error`	0.44	test_sharedfs_lifecycle.py

This is a simple NAS backup plugin for KVM which may be later expanded for other hypervisors. This backup plugin aims to use shared NAS storage on KVM hosts such as NFS (or CephFS and others in future), which is used to backup fully cloned VMs for backup & restore operations. This may NOT be as efficient and performant as some of the other B&R providers, but maybe useful for some KVM environments who are okay to only have full-instance backups and limited functionality. Design & Implementation follows the `networker` B&R plugin, which is simply: - Implement B&R plugin interfaces - Use cmd-answer pattern to execute backup and restore operations on KVM host when VM is running (or needs to be restored) - instead of a B&R API client, relies on answers from KVM agent which executes the operations - Backups are full VM domain snapshots, copied to a VM-specific folders on a NAS target (NFS) along with a domain XML - Backup uses libvirt feature: https://libvirt.org/kbase/live_full_disk_backup.html orchestrated via virsh/bash script (nasbackup.sh) as the libvirt-java lacks the bindings - Supported instance volume storage for restore operations: NFS & local storage Refer the doc PR for feature limitations and usage details: apache/cloudstack-documentation#429 Signed-off-by: Rohit Yadav <[email protected]> Co-authored-by: Pearl Dsilva <[email protected]> Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Suresh Kumar Anaparti <[email protected]>

rohityadavcloud added component:kvm status:work-in-progress component:backup labels Jul 26, 2024

rohityadavcloud added this to the 4.20.0.0 milestone Jul 26, 2024

boring-cyborg bot added the component:build label Jul 26, 2024

rohityadavcloud added 13 commits July 26, 2024 13:09

cleanup and leave TODOs

2604f81

Signed-off-by: Rohit Yadav <[email protected]>

cleanup, refactor; add plugin config keys

b961a4d

Signed-off-by: Rohit Yadav <[email protected]>

WIP B&R tool

9a8bbbb

Signed-off-by: Rohit Yadav <[email protected]>

Merge remote-tracking branch 'origin/main' into kvm-backup-plugin-sha…

aa90b6d

…red-storage

add cmd-answer classes

d5503e7

Signed-off-by: Rohit Yadav <[email protected]>

barebone take backup implementation

f3bc039

Signed-off-by: Rohit Yadav <[email protected]>

prototype backup e2e

f13769d

Signed-off-by: Rohit Yadav <[email protected]>

fix backup script

1933f9e

Signed-off-by: Rohit Yadav <[email protected]>

fix command execution handling

e066c5a

Signed-off-by: Rohit Yadav <[email protected]>

implement backup delete

50e407c

Signed-off-by: Rohit Yadav <[email protected]>

fix nasbackup script

9b0be25

Signed-off-by: Rohit Yadav <[email protected]>

fix op typo

11cbea2

Signed-off-by: Rohit Yadav <[email protected]>

add todos

18e0589

Signed-off-by: Rohit Yadav <[email protected]>

rohityadavcloud assigned Pearl1594 Aug 1, 2024

rohityadavcloud added 2 commits August 2, 2024 12:58

introduce backup repository concept for NAS & other backups

97ce34c

Each backup repository is for a specific zone & provider. Eack NAS bkcp offering is tied to a back repo & each bkup offer assinged to VM -> defined its backup repository Signed-off-by: Rohit Yadav <[email protected]>

complete poc

0b9db75

Signed-off-by: Rohit Yadav <[email protected]>

boring-cyborg bot added the component:api label Aug 2, 2024

rohityadavcloud added 2 commits August 2, 2024 14:27

fix backup script

00b22d4

Signed-off-by: Rohit Yadav <[email protected]>

ui tweaks

7db5178

boring-cyborg bot added the component:UI label Aug 2, 2024

sureshanaparti reviewed Sep 5, 2024

View reviewed changes

engine/schema/src/main/resources/META-INF/db/schema-41910to42000.sql Show resolved Hide resolved

sureshanaparti reviewed Sep 5, 2024

View reviewed changes

engine/schema/src/main/resources/META-INF/db/schema-41910to42000.sql Outdated Show resolved Hide resolved

sureshanaparti reviewed Sep 5, 2024

View reviewed changes

plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java Show resolved Hide resolved

sureshanaparti reviewed Sep 5, 2024

View reviewed changes

server/src/main/java/org/apache/cloudstack/backup/BackupManagerImpl.java Show resolved Hide resolved

sureshanaparti approved these changes Sep 5, 2024

View reviewed changes

rohityadavcloud and others added 4 commits September 5, 2024 16:36

Update engine/schema/src/main/resources/META-INF/db/schema-41910to420…

ee0c5b3

…00.sql Co-authored-by: Suresh Kumar Anaparti <[email protected]>

Update engine/schema/src/main/resources/META-INF/db/schema-41910to420…

42e7319

…00.sql Co-authored-by: Suresh Kumar Anaparti <[email protected]>

add rsync as dependnecy for cloudstack-agent

9bffa4d

Signed-off-by: Rohit Yadav <[email protected]> (cherry picked from commit ecfeb04026e1bbf13dc22bb0aa713677c7708bb8) Signed-off-by: Rohit Yadav <[email protected]>

Merge remote-tracking branch 'origin/main' into kvm-backup-plugin-sha…

11c369e

…red-storage

boring-cyborg bot added component:packaging distro:ubuntu labels Sep 5, 2024

rohityadavcloud added 2 commits September 5, 2024 17:27

Merge branch 'main' into kvm-backup-plugin-shared-storage

7423f5f

fix db schema setup issue

9efdb57

Signed-off-by: Rohit Yadav <[email protected]>

apache deleted a comment from blueorangutan Sep 5, 2024

rajujith approved these changes Sep 5, 2024

View reviewed changes

rohityadavcloud removed the status:needs-testing label Sep 5, 2024

rohityadavcloud merged commit 85765c3 into apache:main Sep 5, 2024
25 of 26 checks passed

Pearl1594 mentioned this pull request Sep 30, 2024

Fix Listing backup schedules #9749

Merged

14 tasks

weizhouapache mentioned this pull request Nov 6, 2024

Veeam B&R : Restore and attach backup volume is not possible #9897

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

backup: simple NAS backup plugin for KVM #9451

backup: simple NAS backup plugin for KVM #9451

rohityadavcloud commented Jul 26, 2024 •

edited

Loading

codecov bot commented Jul 26, 2024 •

edited

Loading

rohityadavcloud commented Aug 1, 2024

rohityadavcloud commented Aug 2, 2024

blueorangutan commented Aug 2, 2024

sureshanaparti left a comment

rajujith left a comment

rohityadavcloud commented Sep 5, 2024

blueorangutan commented Sep 5, 2024

blueorangutan commented Sep 5, 2024

Pearl1594 commented Sep 5, 2024

rohityadavcloud commented Sep 5, 2024

blueorangutan commented Sep 5, 2024

rohityadavcloud commented Sep 5, 2024

blueorangutan commented Sep 6, 2024

blueorangutan commented Sep 6, 2024

backup: simple NAS backup plugin for KVM #9451

backup: simple NAS backup plugin for KVM #9451

Conversation

rohityadavcloud commented Jul 26, 2024 • edited Loading

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots

codecov bot commented Jul 26, 2024 • edited Loading

Codecov Report

rohityadavcloud commented Aug 1, 2024

rohityadavcloud commented Aug 2, 2024

blueorangutan commented Aug 2, 2024

sureshanaparti left a comment

Choose a reason for hiding this comment

rajujith left a comment

Choose a reason for hiding this comment

rohityadavcloud commented Sep 5, 2024

blueorangutan commented Sep 5, 2024

blueorangutan commented Sep 5, 2024

Pearl1594 commented Sep 5, 2024

rohityadavcloud commented Sep 5, 2024

blueorangutan commented Sep 5, 2024

rohityadavcloud commented Sep 5, 2024

blueorangutan commented Sep 6, 2024

blueorangutan commented Sep 6, 2024

rohityadavcloud commented Jul 26, 2024 •

edited

Loading

codecov bot commented Jul 26, 2024 •

edited

Loading