Some Virtual Machine template properties are marked to be updated in-place always

[n0ct1s-k8sh]

When you create a virtual machine template (although I don't know if it happens in standalone VMs), the following properties are marked to be updated in-place always, even after every apply:

• Description.
• IPv4 Addresses.
• IPv6 Addresses.
• Network interface names.
• Disk file format.

This is my shell log when I apply and plan for the first time:

vscode ➜ /workspaces/homelab-infra/pve (main) $ tta
data.proxmox_virtual_environment_datastores.pve_datastores: Reading...
data.proxmox_virtual_environment_version.pve_version: Reading...
data.proxmox_virtual_environment_version.pve_version: Read complete after 0s [id=version]
data.proxmox_virtual_environment_datastores.pve_datastores: Read complete after 0s [id=pve_datastores]

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # module.images.proxmox_virtual_environment_download_file.image_debian_12_20240717 will be created
  + resource "proxmox_virtual_environment_download_file" "image_debian_12_20240717" {
      + checksum            = "9ce1ce8c0f16958dd07bce6dd44d12f4d44d12593432a3a8f7c890c262ce78b0402642fa25c22941760b5a84d631cf81e2cb9dc39815be25bf3a2b56388504c6"
      + checksum_algorithm  = "sha512"
      + content_type        = "iso"
      + datastore_id        = "local"
      + file_name           = "debian-12-generic-amd64-20240717-1811.img"
      + id                  = (known after apply)
      + node_name           = "pve"
      + overwrite           = true
      + overwrite_unmanaged = false
      + size                = (known after apply)
      + upload_timeout      = 600
      + url                 = "https://cloud.debian.org/images/cloud/bookworm/20240717-1811/debian-12-generic-amd64-20240717-1811.qcow2"
      + verify              = true
    }

  # module.templates.proxmox_virtual_environment_vm.vm_template_debian_12_20240717 will be created
  + resource "proxmox_virtual_environment_vm" "vm_template_debian_12_20240717" {
      + acpi                    = true
      + bios                    = "ovmf"
      + boot_order              = [
          + "scsi0",
        ]
      + description             = "Managed by OpenTofu"
      + id                      = (known after apply)
      + ipv4_addresses          = (known after apply)
      + ipv6_addresses          = (known after apply)
      + keyboard_layout         = "es"
      + mac_addresses           = (known after apply)
      + machine                 = "q35"
      + migrate                 = false
      + name                    = "template-debian-12-20240717"
      + network_interface_names = (known after apply)
      + node_name               = "pve"
      + on_boot                 = false
      + protection              = false
      + reboot                  = false
      + scsi_hardware           = "virtio-scsi-pci"
      + stop_on_destroy         = false
      + tablet_device           = true
      + tags                    = [
          + "gnu+linux",
          + "debian12",
        ]
      + template                = true
      + timeout_clone           = 1800
      + timeout_create          = 1800
      + timeout_migrate         = 1800
      + timeout_move_disk       = 1800
      + timeout_reboot          = 1800
      + timeout_shutdown_vm     = 1800
      + timeout_start_vm        = 1800
      + timeout_stop_vm         = 300
      + vm_id                   = (known after apply)

      + agent {
          + enabled = true
          + timeout = "15m"
          + trim    = true
          + type    = "virtio"
        }

      + cpu {
          + architecture = "x86_64"
          + cores        = 1
          + flags        = [
              + "+aes",
              + "+md-clear",
              + "+pcid",
            ]
          + hotplugged   = 0
          + limit        = 0
          + numa         = false
          + sockets      = 1
          + type         = "host"
          + units        = 1024
        }

      + disk {
          + aio               = "io_uring"
          + backup            = true
          + cache             = "none"
          + datastore_id      = "local-lvm"
          + discard           = "on"
          + file_format       = "qcow2"
          + file_id           = (known after apply)
          + interface         = "scsi0"
          + iothread          = true
          + path_in_datastore = (known after apply)
          + replicate         = true
          + size              = 20
          + ssd               = true
        }

      + efi_disk {
          + datastore_id      = "local-lvm"
          + file_format       = (known after apply)
          + pre_enrolled_keys = true
          + type              = "2m"
        }

      + memory {
          + dedicated      = 1024
          + floating       = 0
          + keep_hugepages = false
          + shared         = 0
        }

      + network_device {
          + bridge      = "vmbr0"
          + enabled     = true
          + firewall    = false
          + mac_address = (known after apply)
          + model       = "virtio"
          + mtu         = 0
          + queues      = 0
          + rate_limit  = 0
          + vlan_id     = 0
        }

      + operating_system {
          + type = "l26"
        }

      + serial_device {
          + device = "socket"
        }

      + tpm_state {
          + datastore_id = "local-lvm"
          + version      = "v2.0"
        }

      + vga {
          + memory = 16
          + type   = "std"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + current_version = {
      + release       = "8.2"
      + repository_id = "9355359cd7afbae4"
      + version       = "8.2.2"
    }

Do you want to perform these actions?
  OpenTofu will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.images.proxmox_virtual_environment_download_file.image_debian_12_20240717: Creating...
module.images.proxmox_virtual_environment_download_file.image_debian_12_20240717: Still creating... [10s elapsed]
module.images.proxmox_virtual_environment_download_file.image_debian_12_20240717: Creation complete after 11s [id=local:iso/debian-12-generic-amd64-20240717-1811.img]
module.templates.proxmox_virtual_environment_vm.vm_template_debian_12_20240717: Creating...
module.templates.proxmox_virtual_environment_vm.vm_template_debian_12_20240717: Still creating... [11s elapsed]
module.templates.proxmox_virtual_environment_vm.vm_template_debian_12_20240717: Creation complete after 15s [id=100]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

current_version = {
  "release" = "8.2"
  "repository_id" = "9355359cd7afbae4"
  "version" = "8.2.2"
}
vscode ➜ /workspaces/homelab-infra/pve (main) $ ttp 
data.proxmox_virtual_environment_datastores.pve_datastores: Reading...
data.proxmox_virtual_environment_version.pve_version: Reading...
data.proxmox_virtual_environment_version.pve_version: Read complete after 0s [id=version]
data.proxmox_virtual_environment_datastores.pve_datastores: Read complete after 0s [id=pve_datastores]
module.images.proxmox_virtual_environment_download_file.image_debian_12_20240717: Refreshing state... [id=local:iso/debian-12-generic-amd64-20240717-1811.img]
module.templates.proxmox_virtual_environment_vm.vm_template_debian_12_20240717: Refreshing state... [id=100]

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

OpenTofu will perform the following actions:

  # module.templates.proxmox_virtual_environment_vm.vm_template_debian_12_20240717 will be updated in-place
  ~ resource "proxmox_virtual_environment_vm" "vm_template_debian_12_20240717" {
        id                      = "100"
      ~ ipv4_addresses          = [] -> (known after apply)
      ~ ipv6_addresses          = [] -> (known after apply)
        name                    = "template-debian-12-20240717"
      ~ network_interface_names = [] -> (known after apply)
        tags                    = [
            "debian12",
            "gnu+linux",
        ]
        # (25 unchanged attributes hidden)

      ~ disk {
          ~ file_format       = "raw" -> "qcow2"
            # (12 unchanged attributes hidden)
        }

        # (9 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so OpenTofu can't guarantee to take exactly these actions if you run "tofu apply" now.

Steps to reproduce the behavior:

1. Create a proxmox_virtual_environment_download_file resource to download a qcow2 disk image (file extension renamed to .img as the documentation says in the file_name option).
2. Create a proxmox_virtual_environment_vm resource to create a VM template with the previously downloaded file as the disk.
3. Run tofu apply to create all resources for the first time.
4. Run tofu plan and apply those new changes.
5. Run another tofu plan and the same changes are required to apply again and again.

Please also provide a minimal Terraform configuration that reproduces the issue.

• Image download definition:

resource "proxmox_virtual_environment_download_file" "image_debian_12_20240717" {
  datastore_id = var.iso_datastore
  node_name = var.pve_node_name

  content_type = "iso"
  url = "https://cloud.debian.org/images/cloud/bookworm/20240717-1811/debian-12-generic-amd64-20240717-1811.qcow2"
  checksum_algorithm = "sha512"
  checksum = "9ce1ce8c0f16958dd07bce6dd44d12f4d44d12593432a3a8f7c890c262ce78b0402642fa25c22941760b5a84d631cf81e2cb9dc39815be25bf3a2b56388504c6"
  file_name = "debian-12-generic-amd64-20240717-1811.img"   # Workaround for qcow2 file extension support
}

• VM Template definition:

resource "proxmox_virtual_environment_vm" "vm_template_debian_12_20240717" {
  name = "template-debian-12-20240717"
  description = "Managed by OpenTofu"
  template = true
  tags = ["gnu+linux", "debian12"]

  node_name = var.pve_node_name

  operating_system {
    type = "l26"
  }

  vga {
    type = "std"
  }

  serial_device {}

  boot_order = ["scsi0"]

  on_boot = false

  bios = "ovmf"
  efi_disk {
    datastore_id = var.disk_datastore
    pre_enrolled_keys = true
  }
  tpm_state {
    datastore_id = var.disk_datastore
    version = "v2.0"
  }

  agent {
    enabled = true
    trim = true
    type = "virtio"
  }
  keyboard_layout = "es"
  
  scsi_hardware = "virtio-scsi-pci"
  disk {
    aio = "io_uring"
    backup = true
    cache = "none"
    datastore_id = var.disk_datastore
    discard = "on"
    file_id = var.image_debian_12_20240717
    file_format = "qcow2"
    interface = "scsi0"
    iothread = true
    replicate = true
    size = 20
    ssd = true
  }

  machine = "q35"
  cpu {
    architecture = "x86_64"
    cores = 1
    flags = [
      "+aes",
      "+md-clear",
      "+pcid"
    ]
    type = "host"
  }

  memory {
    dedicated = 1024
  }

  network_device {
    bridge = "vmbr0"
    enabled = true
    model = "virtio"
  }
}

Expected behavior
It is expected to not have any pending changes in the VM resource.

Screenshots

• Single or clustered Proxmox: Single.
• Proxmox version: 8.2.2
• Provider version (ideally it should be the latest version): 0.62.0
• Terraform/OpenTofu version: 1.8.1
• OS (where you run Terraform/OpenTofu from): Debian 12
• Debug logs (TF_LOG=DEBUG terraform apply): Attached in the post.
  apply.log

[bpg]

Hi @n0ct1s-k8sh

From your output:

  ~ resource "proxmox_virtual_environment_vm" "vm_template_debian_12_20240717" {
        id                      = "100"
      ~ ipv4_addresses          = [] -> (known after apply)
      ~ ipv6_addresses          = [] -> (known after apply)
        name                    = "template-debian-12-20240717"
      ~ network_interface_names = [] -> (known after apply)
        tags                    = [
            "debian12",
            "gnu+linux",
        ]
        # (25 unchanged attributes hidden)

      ~ disk {
          ~ file_format       = "raw" -> "qcow2"
            # (12 unchanged attributes hidden)
        }

        # (9 unchanged blocks hidden)
    }

I see only disk.file_format is mishandled and causing the resource update.

ipv4_addresses, ipv6_addresses, network_interface_names are computed attributes, their values are retrieved from the VM, hence (known after apply) status. They are not to be updated in-place (i.e. in the VM), but marked with ~ as potentially to be updated in the TF state with the new values after apply.

[robcxyz]

I was also getting architecture creating a diff to be updated in-place but then after using an image with qemu agent pre-installed the diff went away.

Edit: Actually getting a diff now after adding a few more VMs on the architecture. Not sure why I thought this was resolved before since I remember specifically testing if I got a diff after another apply.

      ~ cpu {
          + architecture = "x86_64"
            # (9 unchanged attributes hidden)
        }

So this definitely seems like a bug.

[windowsrefund]

@bpg While I can appreciate your comment about the values being "potentially" updated by the VM, this does have a bit of a "false positive" bug kinda feel to it. After all, terraform plan is saying it wants to make 1 change. In a professional environment, that's not going to cut it and once the constant explanations wear thin, the condition will just be known and referred to as "that buggy provider's problem". Not saying that's correct... just saying that's the perception when it comes to false positives like this.

After confirming each of these are just empty lists in the state, I guess I'm wondering why they'd need to exist in the state to begin with? Granted, my question relates specifically to a template which has not been created with a initialization block like above.

[GeorgeGedox]

I also always seem to get the architecture to be updated on new machines even when the cpu architecture is specifically set in the config

 # (30 unchanged attributes hidden)

      ~ cpu {
          + architecture = "x86_64"
            # (9 unchanged attributes hidden)
        }

        # (7 unchanged blocks hidden)

[bpg]

@windowsrefund

After all, terraform plan is saying it wants to make 1 change

Yes, this is the actual problem I mentioned regarding the file_format attribute.

After confirming each of these are just empty lists in the state, I guess I'm wondering why they'd need to exist in the state to begin with?

The fact that other attributes like ipv4_addresses are marked as "changed" (~) with the value (known after apply) is expected because those attributes are computed, and their values are not known during the plan. They are in the state because we want to retrieve them after the VM resource creation. But in your case, this is a template, and the VM is not started, so there are no IPs.

There is nothing wrong with the computed attributes; we use quite a lot of them. The problem lies with the "regular" attributes that change value when the provider reads them back from PVE after resource creation (mostly) or update (rarely). These cases are bugs, for sure, and they are present across many resources implemented using the old and deprecated Terraform Plugin SDK. These issues are not always straightforward to fix, as the old SDK does not provide the necessary methods to handle default values for attributes, which is a root cause of these discrepancies.

The long-term plan is to fix all such bugs in the VM resource, as outlined in #1231.

In a professional environment, that's not going to cut it and once the constant explanations wear thin, the condition will just be known and referred to as "that buggy provider's problem".

Agreed, that's not ideal. However, this is a hobby project, and there are only so many hours left in a day after a day job and other commitments. I hope people understand.

[bpg]

@GeorgeGedox this should be fixed in #1524