Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Api keypair restructure #9504

Open
wants to merge 22 commits into
base: main
Choose a base branch
from

Conversation

KlausDornsbach
Copy link
Contributor

@KlausDornsbach KlausDornsbach commented Aug 8, 2024

Description

API access keypairs are primarily used to support interactions between systems, without the need to create sessions (through user and password authentication). Currently, CloudStack's implementation of API keypairs does not allow you to specify permissions for each keypair, simply using the account's default permissions. Additionally, the number of keypairs is limited to one per user and they have no start and end dates.

An extension of the API keypairs functionality was implemented, adding several new features that increase flexibility and security. It is now possible to specify a subset of permissions (from the base account) for each keypair, as well as create more than one key per user. It is also possible to define start and end dates for the validity of a keypair. A key created without an expiration date will always be valid up until it is deleted.

The following endpoints were created:

listUserKeys: Through this API the user will be able to specify a single keypairid to fetch its specific properties, a userid to fetch an user's api keypair list. If no keypairid or userid is provided, the API defaults to fetching information on the calling user. The listall property allows for fetching all keypairs, if not specified, it defaults to false, fetching only the last created apikey.

Name Description Required Default
userid id of the owner of the keypairs no none
keypairid id of keypair no none
listall list all accessable keypairs no false

The API getUserKeys was modified preserving backwards compatibility. It now fetches the last keypair created for the informed user.

The api registerUserKeys was modified so the new API keypair parameters could be specified on creation:

Name Description Required Example Default
id user id Yes b8914774-771e-11ee-8e59-5254003754dc -
name name of the keypair No MyKey userId + " - API Keypair"
startdate date when key becomes valid No 2024-04-09 no start date
enddate date when key expires No 2024-04-09 never expires
description keypair description No read only permissions no description
rules list of API access rules No rules[1].rule=list* rules[1].permission=allow all user API permissions based on Account Role

A new keypair deletion API was added (deleteUserKeys). It will accept only one required argument, the keypair id.

Name Description Required
id api key of the keypair yes

I also added a listUserKeyRules api, allowing the user to list the rules associated with an API keypair.

Name Description Required
id id of the keypair yes

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

API Key Creation and Basic Testing

Single Key (via UI):

  1. I was able to create API keypairs through the UI through the button on the top right of the user view;
  2. Through Cloud Monkey, validated that the keypair had the same permissions as the base user by calling a series of APIs;

Multiple Keys (via Cloud Monkey):

  1. Created multiple keys;
  2. Validated the operation was successful on the DB;
  3. Tested creating a key that was not valid and would become valid in the future, with success;
  4. Tested creating a key that was valid and would become invalid in the future, with success;
  5. Tested trying to create keys that were already invalid and got errors;
  6. All permissions of the API key pairs were consistent with each key pair tested.

Permissions Validation

Tested the permissions of keyrules listing, keypair listing, keypair deletion and keypair cretion with the following user/account/domain setup:

  • domain /ROOT
    • account root admin

      • user root admin
      • user user1
    • domain subdomain

      • account domain admin
        • user domain admin
      • account userAccount
        • user user2
        • user user3

The following table describes the results obtained when the user on the first column attempted to operate on the keypairs of users on the first row (V: operation was possible, F: operation was not possible).

user rootAdm user1 domainAdm user2 user3
rootAdm V V V V V
user1 V V V V V
domainAdm F F V V V
user2 F F F V F
user3 F F F F V

Migration to api_keypair Table

  1. A key was created for a read-only user
  2. The database was updated from version 4.19 to 4.20
  3. The API key data was successfully migrated to the api_keypair table, with the corresponding columns in the user table deleted.
  4. Confirmed correct permission handling of the api key.

General validations

  • Could not create an API keypair with more permissions than the base keypair.
  • Deleting system keys was not possible.
  • After deleting a user or account, the API keypairs were invalidated.

Copy link

codecov bot commented Aug 12, 2024

Codecov Report

Attention: Patch coverage is 18.00220% with 747 lines in your changes missing coverage. Please review.

Project coverage is 15.82%. Comparing base (b19c069) to head (ef69e83).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...he/cloudstack/api/response/ApiKeyPairResponse.java 0.00% 109 Missing ⚠️
...n/java/org/apache/cloudstack/acl/ApiKeyPairVO.java 26.77% 92 Missing and 1 partial ⚠️
...c/main/java/com/cloud/user/AccountManagerImpl.java 49.13% 78 Missing and 10 partials ⚠️
...ck/api/command/admin/user/RegisterUserKeysCmd.java 0.00% 80 Missing ⚠️
...g/apache/cloudstack/acl/dao/ApiKeyPairDaoImpl.java 0.00% 64 Missing ⚠️
...src/main/java/com/cloud/api/ApiResponseHelper.java 0.00% 46 Missing ⚠️
...ava/com/cloud/upgrade/dao/Upgrade41910to42000.java 2.63% 37 Missing ⚠️
...tack/api/command/admin/user/DeleteUserKeysCmd.java 0.00% 31 Missing ⚠️
...oudstack/acl/dao/ApiKeyPairPermissionsDaoImpl.java 0.00% 26 Missing ⚠️
...g/apache/cloudstack/acl/ApiKeyPairManagerImpl.java 25.71% 24 Missing and 2 partials ⚠️
... and 17 more
Additional details and impacted files
@@             Coverage Diff             @@
##               main    #9504     +/-   ##
===========================================
  Coverage     15.81%   15.82%             
- Complexity    12554    12572     +18     
===========================================
  Files          5629     5638      +9     
  Lines        492028   492737    +709     
  Branches      62750    61359   -1391     
===========================================
+ Hits          77813    77970    +157     
- Misses       405892   406429    +537     
- Partials       8323     8338     +15     
Flag Coverage Δ
uitests 4.48% <ø> (ø)
unittests 16.61% <18.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@DaanHoogland
Copy link
Contributor

nice feature @KlausDornsbach , some suggestions,

  1. would it make sense to be able to delete a kay based on name?
  2. also is admin allowed to delete keys for other users? (would make sense from a maintainance point of view, would it?)

@KlausDornsbach
Copy link
Contributor Author

Hey @DaanHoogland, thanks for taking a look!

It would make sense to be able to delete a keypair by name, we would just need to block users from creating multiple API keypairs with the same name.

At the moment an admin is allowed to delete keypairs from users it has access, for example, a root admin user could delete any keypair in the platform, domain admin users can delete any keypair in the domain, normal users can only delete their own keys. These permissions are also true for visualization and creation APIs.

@DaanHoogland
Copy link
Contributor

It would make sense to be able to delete a keypair by name, we would just need to block users from creating multiple API keypairs with the same name.

Well, I think a unique constraint on UserId/KeyPairName makes sense also from a usability sense.

At the moment an admin is allowed to delete keypairs from users it has access, for example, a root admin user could delete any keypair in the platform, domain admin users can delete any keypair in the domain, normal users can only delete their own keys. These permissions are also true for visualization and creation APIs.

👍

@rajujith rajujith self-assigned this Aug 19, 2024
@rajujith
Copy link
Collaborator

@blueorangutan package

Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@apache apache deleted a comment from blueorangutan Aug 20, 2024
@apache apache deleted a comment from blueorangutan Aug 20, 2024
@apache apache deleted a comment from blueorangutan Aug 20, 2024
@DaanHoogland
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10869

@DaanHoogland
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-11249)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 1307 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9504-t11249-kvm-ol8.zip
Smoke tests completed. 0 look OK, 0 have errors, 139 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
all_test_2fa Skipped --- test_2fa.py
all_test_account_access Skipped --- test_account_access.py
all_test_accounts Skipped --- test_accounts.py
all_test_affinity_groups_projects Skipped --- test_affinity_groups_projects.py
all_test_affinity_groups Skipped --- test_affinity_groups.py
all_test_annotations Skipped --- test_annotations.py
all_test_async_job Skipped --- test_async_job.py
all_test_attach_multiple_volumes Skipped --- test_attach_multiple_volumes.py
all_test_backup_recovery_dummy Skipped --- test_backup_recovery_dummy.py
all_test_backup_recovery_veeam Skipped --- test_backup_recovery_veeam.py
all_test_bucket Skipped --- test_bucket.py
all_test_certauthority_root Skipped --- test_certauthority_root.py
all_test_cluster_drs Skipped --- test_cluster_drs.py
all_test_console_endpoint Skipped --- test_console_endpoint.py
all_test_create_list_domain_account_project Skipped --- test_create_list_domain_account_project.py
all_test_create_network Skipped --- test_create_network.py
all_test_deploy_vgpu_enabled_vm Skipped --- test_deploy_vgpu_enabled_vm.py
all_test_deploy_virtio_scsi_vm Skipped --- test_deploy_virtio_scsi_vm.py
all_test_deploy_vm_extra_config_data Skipped --- test_deploy_vm_extra_config_data.py
all_test_deploy_vm_iso Skipped --- test_deploy_vm_iso.py
all_test_deploy_vm_iso_uefi Skipped --- test_deploy_vm_iso_uefi.py
all_test_deploy_vm_root_resize Skipped --- test_deploy_vm_root_resize.py
all_test_deploy_vms_in_parallel Skipped --- test_deploy_vms_in_parallel.py
all_test_deploy_vms_with_varied_deploymentplanners Skipped --- test_deploy_vms_with_varied_deploymentplanners.py
all_test_deploy_vm_with_userdata Skipped --- test_deploy_vm_with_userdata.py
all_test_diagnostics Skipped --- test_diagnostics.py
all_test_direct_download Skipped --- test_direct_download.py
all_test_disk_offerings Skipped --- test_disk_offerings.py
all_test_disk_provisioning_types Skipped --- test_disk_provisioning_types.py
all_test_domain_disk_offerings Skipped --- test_domain_disk_offerings.py
all_test_domain_network_offerings Skipped --- test_domain_network_offerings.py
all_test_domain_service_offerings Skipped --- test_domain_service_offerings.py
all_test_domain_vpc_offerings Skipped --- test_domain_vpc_offerings.py
all_test_dynamicroles Skipped --- test_dynamicroles.py
all_test_enable_account_settings_for_domain Skipped --- test_enable_account_settings_for_domain.py
all_test_enable_role_based_users_in_projects Skipped --- test_enable_role_based_users_in_projects.py
all_test_events_resource Skipped --- test_events_resource.py
all_test_gateway_on_shared_networks Skipped --- test_gateway_on_shared_networks.py
all_test_global_acls Skipped --- test_global_acls.py
all_test_global_settings Skipped --- test_global_settings.py
all_test_guest_os Skipped --- test_guest_os.py
all_test_guest_vlan_range Skipped --- test_guest_vlan_range.py
all_test_host_control_state Skipped --- test_host_control_state.py
all_test_hostha_simulator Skipped --- test_hostha_simulator.py
all_test_host_ping Skipped --- test_host_ping.py
all_test_host_tags Skipped --- test_host_tags.py
all_test_human_readable_logs Skipped --- test_human_readable_logs.py
all_test_image_store_object_migration Skipped --- test_image_store_object_migration.py
all_test_import_unmanage_volumes Skipped --- test_import_unmanage_volumes.py
all_test_internal_lb Skipped --- test_internal_lb.py
all_test_ipv6_infra Skipped --- test_ipv6_infra.py
all_test_iso Skipped --- test_iso.py
all_test_kubernetes_clusters Skipped --- test_kubernetes_clusters.py
all_test_kubernetes_supported_versions Skipped --- test_kubernetes_supported_versions.py
all_test_list_accounts Skipped --- test_list_accounts.py
all_test_list_disk_offerings Skipped --- test_list_disk_offerings.py
all_test_list_domains Skipped --- test_list_domains.py
all_test_list_hosts Skipped --- test_list_hosts.py
all_test_list_ids_parameter Skipped --- test_list_ids_parameter.py
all_test_list_service_offerings Skipped --- test_list_service_offerings.py
all_test_list_storage_pools Skipped --- test_list_storage_pools.py
all_test_list_volumes Skipped --- test_list_volumes.py
all_test_loadbalance Skipped --- test_loadbalance.py
all_test_login Skipped --- test_login.py
all_test_metrics_api Skipped --- test_metrics_api.py
all_test_migration Skipped --- test_migration.py
all_test_multipleips_per_nic Skipped --- test_multipleips_per_nic.py
all_test_nested_virtualization Skipped --- test_nested_virtualization.py
all_test_network_acl Skipped --- test_network_acl.py
all_test_network_ipv6 Skipped --- test_network_ipv6.py
all_test_network_permissions Skipped --- test_network_permissions.py
all_test_network Skipped --- test_network.py
all_test_nic_adapter_type Skipped --- test_nic_adapter_type.py
all_test_nic Skipped --- test_nic.py
all_test_non_contigiousvlan Skipped --- test_non_contigiousvlan.py
all_test_nonstrict_affinity_group Skipped --- test_nonstrict_affinity_group.py
all_test_object_stores Skipped --- test_object_stores.py
all_test_outofbandmanagement_nestedplugin Skipped --- test_outofbandmanagement_nestedplugin.py
all_test_outofbandmanagement Skipped --- test_outofbandmanagement.py
all_test_over_provisioning Skipped --- test_over_provisioning.py
all_test_password_server Skipped --- test_password_server.py
all_test_persistent_network Skipped --- test_persistent_network.py
all_test_portable_publicip Skipped --- test_portable_publicip.py
all_test_portforwardingrules Skipped --- test_portforwardingrules.py
all_test_primary_storage Skipped --- test_primary_storage.py
all_test_primary_storage_scope Skipped --- test_primary_storage_scope.py
all_test_privategw_acl_ovs_gre Skipped --- test_privategw_acl_ovs_gre.py
all_test_privategw_acl Skipped --- test_privategw_acl.py
all_test_private_roles Skipped --- test_private_roles.py
all_test_projects Skipped --- test_projects.py
all_test_public_ip_range Skipped --- test_public_ip_range.py
all_test_purge_expunged_vms Skipped --- test_purge_expunged_vms.py
all_test_pvlan Skipped --- test_pvlan.py
all_test_quarantined_ips Skipped --- test_quarantined_ips.py
all_test_regions Skipped --- test_regions.py
all_test_register_userdata Skipped --- test_register_userdata.py
all_test_reset_configuration_settings Skipped --- test_reset_configuration_settings.py
all_test_reset_vm_on_reboot Skipped --- test_reset_vm_on_reboot.py
all_test_resource_accounting Skipped --- test_resource_accounting.py
all_test_resource_detail Skipped --- test_resource_detail.py
all_test_resource_names Skipped --- test_resource_names.py
all_test_restore_vm Skipped --- test_restore_vm.py
all_test_router_dhcphosts Skipped --- test_router_dhcphosts.py
all_test_router_dns Skipped --- test_router_dns.py
all_test_router_dnsservice Skipped --- test_router_dnsservice.py
all_test_routers_iptables_default_policy Skipped --- test_routers_iptables_default_policy.py
all_test_routers_network_ops Skipped --- test_routers_network_ops.py
all_test_routers Skipped --- test_routers.py
all_test_safe_shutdown Skipped --- test_safe_shutdown.py
all_test_scale_vm Skipped --- test_scale_vm.py
all_test_secondary_storage Skipped --- test_secondary_storage.py
all_test_service_offerings Skipped --- test_service_offerings.py
all_test_set_sourcenat Skipped --- test_set_sourcenat.py
all_test_snapshots Skipped --- test_snapshots.py
all_test_ssvm Skipped --- test_ssvm.py
all_test_staticroles Skipped --- test_staticroles.py
all_test_storage_policy Skipped --- test_storage_policy.py
all_test_templates Skipped --- test_templates.py
all_test_update_security_group Skipped --- test_update_security_group.py
all_test_usage_events Skipped --- test_usage_events.py
all_test_usage Skipped --- test_usage.py
all_test_vm_autoscaling Skipped --- test_vm_autoscaling.py
all_test_vm_deployment_planner Skipped --- test_vm_deployment_planner.py
all_test_vm_life_cycle Skipped --- test_vm_life_cycle.py
all_test_vm_lifecycle_unmanage_import Skipped --- test_vm_lifecycle_unmanage_import.py
all_test_vm_schedule Skipped --- test_vm_schedule.py
all_test_vm_snapshot_kvm Skipped --- test_vm_snapshot_kvm.py
all_test_vm_snapshots Skipped --- test_vm_snapshots.py
all_test_vm_strict_host_tags Skipped --- test_vm_strict_host_tags.py
all_test_vnf_templates Skipped --- test_vnf_templates.py
all_test_volumes Skipped --- test_volumes.py
all_test_vpc_ipv6 Skipped --- test_vpc_ipv6.py
all_test_vpc_redundant Skipped --- test_vpc_redundant.py
all_test_vpc_router_nics Skipped --- test_vpc_router_nics.py
all_test_vpc_vpn Skipped --- test_vpc_vpn.py
all_test_webhook_delivery Skipped --- test_webhook_delivery.py
all_test_webhook_lifecycle Skipped --- test_webhook_lifecycle.py
all_test_host_maintenance Skipped --- test_host_maintenance.py
all_test_hostha_kvm Skipped --- test_hostha_kvm.py

@DaanHoogland
Copy link
Contributor

@KlausDornsbach all the tests were skipped because of errors like these

2024-08-30 08:17:15,447 - ERROR - Exception:['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/site-packages/marvin/cloudstackConnection.py", line 310, in __parseAndGetResponse\n    response_cls)\n', '  File "/usr/local/lib/python3.6/site-packages/marvin/jsonHelper.py", line 155, in getResultObj\n    raise cloudstackException.CloudstackAPIException(respname, errMsg)\n', 'marvin.cloudstackException.CloudstackAPIException: Execute cmd: getuserkeys failed, due to: errorCode: 530, errorText:Cannot invoke "org.apache.cloudstack.acl.apikeypair.ApiKeyPair.getApiKey()" because "keyPair" is null\n']
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/marvin/cloudstackConnection.py", line 310, in __parseAndGetResponse
    response_cls)
  File "/usr/local/lib/python3.6/site-packages/marvin/jsonHelper.py", line 155, in getResultObj
    raise cloudstackException.CloudstackAPIException(respname, errMsg)
marvin.cloudstackException.CloudstackAPIException: Execute cmd: getuserkeys failed, due to: errorCode: 530, errorText:Cannot invoke "org.apache.cloudstack.acl.apikeypair.ApiKeyPair.getApiKey()" because "keyPair" is null

can you explain this as a bug or feature somehow? It seems most of the simulator tests (from the github actions) passed but none of the tests with hypervisors.

Copy link

github-actions bot commented Sep 5, 2024

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@DaanHoogland
Copy link
Contributor

@KlausDornsbach , can you look at the conflicts and the comments please?

@KlausDornsbach
Copy link
Contributor Author

Sorry for taking a bit long @DaanHoogland!
Hopefully the issue with the code is fixed. Still need to set up marvin locally.

@rajujith
Copy link
Collaborator

rajujith commented Sep 9, 2024

@blueorangutan package

@blueorangutan
Copy link

@rajujith a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11049

Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@DaanHoogland
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11120

@DaanHoogland
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-11485)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 64141 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9504-t11485-kvm-ol8.zip
Smoke tests completed. 138 look OK, 1 have errors, 2 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_user_key_renew_same_account Error 1.58 test_accounts.py
all_test_host_maintenance Skipped --- test_host_maintenance.py
all_test_hostha_kvm Skipped --- test_hostha_kvm.py

@DaanHoogland DaanHoogland added this to the 4.21.0.0 milestone Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants