api,agent,server,engine-schema: scalability improvements #9840

shwstppr · 2024-10-23T10:05:27Z

Description

Following changes and improvements have been added:

Improvements in handling of PingRoutingCommand
1. Added global config - vm.sync.power.state.transitioning, default value: true, to control syncing of power states for transitioning VMs. This can be set to false to prevent computation of transitioning state VMs.
2. Improved VirtualMachinePowerStateSync to allow power state sync for host VMs in a batch
3. Optimized scanning stalled VMs
Added option to set worker threads for capacity calculation using config - capacity.calculate.workers
Added caching for account/use role API access with expiration after write can be configured using config - dynamic.apichecker.cache.period. If set to zero then there will be no caching. Default is 0.
Added caching for account/use role API access with expiration after write set to 60 seconds.
Added caching for some recurring DB retrievals
1. CapacityManager - listing service offerings - beneficial in host capacity calculation
2. LibvirtServerDiscoverer existing host for the cluster - beneficial for host joins
3. DownloadListener - hypervisors for zone - beneficial for host joins
4. VirtualMachineManagerImpl - VMs in progress- beneficial for processing stalled VMs during PingRoutingCommands
Optimized MS list retrieval for agent connect
Optimize finding ready systemvm template for zone
Database retrieval optimisations - fix and refactor for cases where only IDs or counts are used mainly for hosts and other infra entities. Also similar cases for VMs and other entities related to host concerning background tasks
Changes in agent-agentmanager connection with NIO client-server classes
1. Optimized the use of the executor service
2. Refactore Agent class to better handle connections.
3. Do SSL handshakes within worker threads
4. Added global configs to control the behaviour depending on the infra. SSL handshake could be a bottleneck during agent connections. Configs - agent.ssl.handshake.min.workers and agent.ssl.handshake.max.workers can be used to control number of new connections management server handles at a time. agent.ssl.handshake.timeout can be used to set number of seconds after which SSL handshake times out at MS end.
5. On agent side backoff and sslhandshake timeout can be controlled by agent properties. backoff.seconds and ssl.handshake.timeout properties can be used.
Improvements in StatsCollection - minimize DB retrievals.
Improvements in DeploymentPlanner allow for the retrieval of only desired host fields and fewer retrievals.
Improvements in hosts connection for a storage pool. Added config - storage.pool.host.connect.workers to control the number of worker threads that can be used to connect hosts to a storage pool. Worker thread approach is followed currently only for NFS and ScaleIO pools.
Minor improvements in resource limit calculations wrt DB retrievals

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2024-10-23T10:12:57Z

Codecov Report

Attention: Patch coverage is 19.30116% with 1455 lines in your changes missing coverage. Please review.

Project coverage is 15.60%. Comparing base (019f2c6) to head (8541823).
Report is 26 commits behind head on main.

Files with missing lines	Patch %	Lines
agent/src/main/java/com/cloud/agent/Agent.java	20.86%	264 Missing and 9 partials ⚠️
.../src/main/java/com/cloud/host/dao/HostDaoImpl.java	0.00%	158 Missing ⚠️
.../main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java	11.84%	131 Missing and 3 partials ⚠️
.../apache/cloudstack/metrics/MetricsServiceImpl.java	0.00%	73 Missing and 1 partial ⚠️
...n/java/com/cloud/capacity/CapacityManagerImpl.java	20.45%	70 Missing ⚠️
...com/cloud/vm/VirtualMachinePowerStateSyncImpl.java	0.00%	67 Missing ⚠️
...n/java/com/cloud/vm/VirtualMachineManagerImpl.java	0.00%	60 Missing ⚠️
...rc/main/java/com/cloud/alert/AlertManagerImpl.java	0.00%	51 Missing ⚠️
...src/main/java/com/cloud/server/StatsCollector.java	0.00%	47 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	30.00%	39 Missing and 3 partials ⚠️
... and 54 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9840      +/-   ##
============================================
- Coverage     15.78%   15.60%   -0.19%     
- Complexity    12564    12624      +60     
============================================
  Files          5627     5631       +4     
  Lines        492250   492895     +645     
  Branches      61405    59709    -1696     
============================================
- Hits          77710    76906     -804     
- Misses       406066   407490    +1424     
- Partials       8474     8499      +25

Flag	Coverage Δ
uitests	`4.04% <ø> (+<0.01%)`	⬆️
unittests	`16.41% <19.30%> (-0.20%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Following changes and improvements have been added: - Improvements in handling of PingRoutingCommand 1. Added global config - `vm.sync.power.state.transitioning`, default value: true, to control syncing of power states for transitioning VMs. This can be set to false to prevent computation of transitioning state VMs. 2. Improved VirtualMachinePowerStateSync to allow power state sync for host VMs in a batch 3. Optimized scanning stalled VMs - Added option to set worker threads for capacity calculation using config - `capacity.calculate.workers` - Added caching framework based on Caffeine in-memory caching library, https://github.com/ben-manes/caffeine - Added caching for account/use role API access with expiration after write can be configured using config - `dynamic.apichecker.cache.period`. If set to zero then there will be no caching. Default is 0. - Added caching for account/use role API access with expiration after write set to 60 seconds. - Added caching for some recurring DB retrievals 1. CapacityManager - listing service offerings - beneficial in host capacity calculation 2. LibvirtServerDiscoverer existing host for the cluster - beneficial for host joins 3. DownloadListener - hypervisors for zone - beneficial for host joins 5. VirtualMachineManagerImpl - VMs in progress- beneficial for processing stalled VMs during PingRoutingCommands - Optimized MS list retrieval for agent connect - Optimize finding ready systemvm template for zone - Database retrieval optimisations - fix and refactor for cases where only IDs or counts are used mainly for hosts and other infra entities. Also similar cases for VMs and other entities related to host concerning background tasks - Changes in agent-agentmanager connection with NIO client-server classes 1. Optimized the use of the executor service 2. Refactore Agent class to better handle connections. 3. Do SSL handshakes within worker threads 5. Added global configs to control the behaviour depending on the infra. SSL handshake could be a bottleneck during agent connections. Configs - `agent.ssl.handshake.min.workers` and `agent.ssl.handshake.max.workers` can be used to control number of new connections management server handles at a time. `agent.ssl.handshake.timeout` can be used to set number of seconds after which SSL handshake times out at MS end. 6. On agent side backoff and sslhandshake timeout can be controlled by agent properties. `backoff.seconds` and `ssl.handshake.timeout` properties can be used. - Improvements in StatsCollection - minimize DB retrievals. - Improvements in DeploymentPlanner allow for the retrieval of only desired host fields and fewer retrievals. - Improvements in hosts connection for a storage pool. Added config - `storage.pool.host.connect.workers` to control the number of worker threads that can be used to connect hosts to a storage pool. Worker thread approach is followed currently only for NFS and ScaleIO pools. - Minor improvements in resource limit calculations wrt DB retrievals Signed-off-by: Abhishek Kumar <[email protected]> Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Rohit Yadav <[email protected]>

GutoVeronezi · 2024-10-23T13:31:18Z

Honestly, I don't like PRs with thousand of lines doing thousand of things. It is hard to review and test. I encourage you to separate it in several minor PRs that address each one of the changes you are proposing.

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2024-10-25T10:46:38Z

@blueorangutan package

blueorangutan · 2024-10-25T10:48:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-10-25T11:58:31Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11441

shwstppr · 2024-11-02T16:23:47Z

@blueorangutan package

blueorangutan · 2024-11-02T16:26:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-02T17:41:03Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11486

shwstppr · 2024-11-02T17:43:52Z

@blueorangutan test

blueorangutan · 2024-11-02T17:46:03Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-11-03T14:20:15Z

[SF] Trillian test result (tid-11738)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 71834 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9840-t11738-kvm-ol8.zip
Smoke tests completed. 134 look OK, 6 have errors, 1 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_host_tags	`Failure`	10.40	test_host_tags.py
test_03_secured_to_nonsecured_vm_migration	`Error`	1286.75	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	1286.76	test_vm_life_cycle.py
ContextSuite context=TestSecuredVmMigration>:teardown	`Error`	2253.98	test_vm_life_cycle.py
ContextSuite context=TestVAppsVM>:setup	`Error`	2409.13	test_vm_life_cycle.py
ContextSuite context=TestVMLifeCycle>:setup	`Error`	2499.28	test_vm_life_cycle.py
ContextSuite context=TestVMSchedule>:setup	`Error`	0.00	test_vm_schedule.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	2101.84	test_hostha_kvm.py
test_01_migrate_vm_strict_tags_success	`Error`	99.22	test_vm_strict_host_tags.py
test_01_redundant_vpc_site2site_vpn	`Failure`	455.29	test_vpc_vpn.py
all_test_vm_lifecycle_unmanage_import	`Skipped`	---	test_vm_lifecycle_unmanage_import.py

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2024-11-04T11:40:53Z

@blueorangutan package

blueorangutan · 2024-11-04T11:42:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache

great job @shwstppr

over lgtm

it seems this will reduce a large number of database queries, therefore improve the performance a lot.

agent/conf/agent.properties

agent/src/main/java/com/cloud/agent/Agent.java

engine/schema/src/main/java/com/cloud/storage/dao/StoragePoolHostDaoImpl.java

server/src/main/java/com/cloud/storage/StorageManagerImpl.java

server/src/main/java/org/apache/cloudstack/acl/RoleManagerImpl.java

tools/marvin/setup.py

utils/src/main/java/com/cloud/utils/nio/NioConnection.java

blueorangutan · 2024-11-04T12:46:20Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11496

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2024-11-05T09:07:07Z

Thanks @weizhouapache for the review. I've made the changes as per your suggestions and responded to the queries.

@blueorangutan package

blueorangutan · 2024-11-05T09:08:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-05T10:20:13Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11505

shwstppr · 2024-11-05T18:23:36Z

@blueorangutan test

blueorangutan · 2024-11-05T18:24:03Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

weizhouapache · 2024-11-06T11:59:09Z

@blueorangutan test

blueorangutan · 2024-11-06T12:00:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-11-06T12:53:18Z

[SF] Trillian test result (tid-11749)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 64218 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9840-t11749-kvm-ol8.zip
Smoke tests completed. 136 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_cpvm_vm	`Failure`	0.03	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.03	test_ssvm.py
test_01_migrate_VM_and_root_volume	`Error`	100.30	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	54.63	test_vm_life_cycle.py
test_08_migrate_vm	`Error`	50.01	test_vm_life_cycle.py
test_01_migrate_vm_strict_tags_success	`Error`	105.19	test_vm_strict_host_tags.py
test_01_verify_ipv6_vpc	`Error`	1704.96	test_vpc_ipv6.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	622.70	test_vpc_redundant.py

blueorangutan · 2024-11-07T04:33:16Z

[SF] Trillian test result (tid-11754)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 57015 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9840-t11754-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_sys_vm_start	`Failure`	0.13	test_secondary_storage.py

harikrishna-patnala

minor comment, overall LGTM. Nice work @shwstppr

harikrishna-patnala · 2024-11-07T07:26:43Z

engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade41910to42000.java

@@ -58,6 +58,7 @@ public InputStream[] getPrepareScripts() {
    @Override


Making a note to change this file after 4.20 release

Signed-off-by: Abhishek Kumar <[email protected]>

shwstppr · 2024-11-12T07:30:22Z

Thank you @harikrishna-patnala for the review

@GutoVeronezi thank you for the feedback and suggestion. I agree with the rationale of smaller PRs but I would request you and others to consider this as an exception. The work in this PR is a part of larger scalability work which I'll also be presenting at CCC. In this PR, the focus is mainly on persistence layer and infrastructure entities. There are also some changes wrt schema and other areas which were quick fixes and yield performance improvements. I'll be adding more details after the conference, including some benchmarking tests and unit tests.
If you've reservations around a particular set of changes or module, I can try creating a separate diff for that and discuss creating a separate PR around that.

shwstppr · 2024-11-12T07:32:10Z

@blueorangutan package

blueorangutan · 2024-11-12T07:34:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-12T08:40:49Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11549

boring-cyborg bot added component:agent component:api component:compute component:orchestration labels Oct 23, 2024

shwstppr force-pushed the scalability-improvements-changes branch from 2c750db to 080e5af Compare October 23, 2024 12:52

shwstppr force-pushed the scalability-improvements-changes branch from 080e5af to e3cf7fd Compare October 23, 2024 13:27

JoaoJandre requested review from GutoVeronezi, BryanMLima and JoaoJandre October 23, 2024 13:28

shwstppr added 4 commits October 24, 2024 10:22

test1, domaindetails, capacitymanager fix

feec5b1

Signed-off-by: Abhishek Kumar <[email protected]>

test2 - agent tests

eafd03c

Signed-off-by: Abhishek Kumar <[email protected]>

capacitymanagertest fix

5159f26

Signed-off-by: Abhishek Kumar <[email protected]>

change

4ffb091

Signed-off-by: Abhishek Kumar <[email protected]>

apache deleted a comment from blueorangutan Oct 25, 2024

fix missing changes

779d521

Signed-off-by: Abhishek Kumar <[email protected]>

weizhouapache approved these changes Nov 4, 2024

View reviewed changes

shwstppr added 4 commits November 5, 2024 13:18

address comments

7d43ce3

Signed-off-by: Abhishek Kumar <[email protected]>

revert marvin/setup.py

a076bad

Signed-off-by: Abhishek Kumar <[email protected]>

fix indent

f63fcea

Signed-off-by: Abhishek Kumar <[email protected]>

use space in sql

da477f6

Signed-off-by: Abhishek Kumar <[email protected]>

harikrishna-patnala approved these changes Nov 7, 2024

View reviewed changes

shwstppr added 3 commits November 12, 2024 11:52

address duplicate

e36c6a5

Signed-off-by: Abhishek Kumar <[email protected]>

update host logs

305f954

Signed-off-by: Abhishek Kumar <[email protected]>

revert e36c6a5

8541823

Signed-off-by: Abhishek Kumar <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

api,agent,server,engine-schema: scalability improvements #9840

api,agent,server,engine-schema: scalability improvements #9840

shwstppr commented Oct 23, 2024 •

edited

Loading

codecov bot commented Oct 23, 2024 •

edited

Loading

GutoVeronezi commented Oct 23, 2024

shwstppr commented Oct 25, 2024

blueorangutan commented Oct 25, 2024

blueorangutan commented Oct 25, 2024

shwstppr commented Nov 2, 2024

blueorangutan commented Nov 2, 2024

blueorangutan commented Nov 2, 2024

shwstppr commented Nov 2, 2024

blueorangutan commented Nov 2, 2024

blueorangutan commented Nov 3, 2024

shwstppr commented Nov 4, 2024

blueorangutan commented Nov 4, 2024

weizhouapache left a comment

blueorangutan commented Nov 4, 2024

shwstppr commented Nov 5, 2024

blueorangutan commented Nov 5, 2024

blueorangutan commented Nov 5, 2024

shwstppr commented Nov 5, 2024

blueorangutan commented Nov 5, 2024

weizhouapache commented Nov 6, 2024

blueorangutan commented Nov 6, 2024

blueorangutan commented Nov 6, 2024

blueorangutan commented Nov 7, 2024

harikrishna-patnala left a comment

harikrishna-patnala Nov 7, 2024

shwstppr commented Nov 12, 2024

shwstppr commented Nov 12, 2024

blueorangutan commented Nov 12, 2024

blueorangutan commented Nov 12, 2024

		@@ -58,6 +58,7 @@ public InputStream[] getPrepareScripts() {
		@Override

api,agent,server,engine-schema: scalability improvements #9840

Are you sure you want to change the base?

api,agent,server,engine-schema: scalability improvements #9840

Conversation

shwstppr commented Oct 23, 2024 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov bot commented Oct 23, 2024 • edited Loading

Codecov Report

GutoVeronezi commented Oct 23, 2024

shwstppr commented Oct 25, 2024

blueorangutan commented Oct 25, 2024

blueorangutan commented Oct 25, 2024

shwstppr commented Nov 2, 2024

blueorangutan commented Nov 2, 2024

blueorangutan commented Nov 2, 2024

shwstppr commented Nov 2, 2024

blueorangutan commented Nov 2, 2024

blueorangutan commented Nov 3, 2024

shwstppr commented Nov 4, 2024

blueorangutan commented Nov 4, 2024

weizhouapache left a comment

Choose a reason for hiding this comment

blueorangutan commented Nov 4, 2024

shwstppr commented Nov 5, 2024

blueorangutan commented Nov 5, 2024

blueorangutan commented Nov 5, 2024

shwstppr commented Nov 5, 2024

blueorangutan commented Nov 5, 2024

weizhouapache commented Nov 6, 2024

blueorangutan commented Nov 6, 2024

blueorangutan commented Nov 6, 2024

blueorangutan commented Nov 7, 2024

harikrishna-patnala left a comment

Choose a reason for hiding this comment

harikrishna-patnala Nov 7, 2024

Choose a reason for hiding this comment

shwstppr commented Nov 12, 2024

shwstppr commented Nov 12, 2024

blueorangutan commented Nov 12, 2024

blueorangutan commented Nov 12, 2024

shwstppr commented Oct 23, 2024 •

edited

Loading

codecov bot commented Oct 23, 2024 •

edited

Loading