Recursive references or co-dependent resources?

[jsblake]

I have a situation that I don't think is unique, and I'm curious if others have run into this scenario and if so what their solution is, or, just some input on my solution.

The scenario is this: I am deploying a vnet and a set of VMs. These VMs will include a traditional Active Directory domain controller, and the other VMs will be joined to the domain. So, I need the DNS of the vnet to point towards the IP of the VM serving as the domain controller

The issue here is that I don't want to statically assign IPs to VMs before creation, as I am creating many of these environments and they occupy varying IP spaces. Here comes the problem:

How do you set the DNS of the vnet to the IP of a VM that depends on the vnet?

My solution in code is below. Basically, I define the vnet, and then define the VM depending on the initial vnet creation. I then define the vnet a second time, with the DNS property set to the IP address output from the VM creation, and that second vnet definition is set to depend on the VM

This works, but seems less than perfect. Maybe it's the only way, but curious if anyone has a better way? Or at least this could serve as feedback to the team that there are these recursive reference cases out there

// this is all in the same template file
//first vnet resource actually creates the vnet
resource virtualNetwork 'Microsoft.Network/virtualNetworks@2019-11-01' = {
  name: vnetName
  location: resourceGroup().location
  properties: {
    addressSpace: {
      addressPrefixes: [
        vnetCIDR
      ]
    }
    subnets: [
      {
        name: subnetName
        properties: {
          networkSecurityGroup: {
            id: nsg.id          
          }
          addressPrefix: subnetCIDR
        }
      }
    ]
  }
}

// the VM's network interface depends on the first vnet definition by reference in the ipconfig
resource networkInterface 'Microsoft.Network/networkInterfaces@2020-11-01' = {
  name: 'virtualmachine-nic'
  location: resourceGroup().location
  properties: {
    enableAcceleratedNetworking: false
    ipConfigurations: [
      {
        name: 'ipConfig1'
        properties: {
          privateIPAllocationMethod: 'Dynamic'
          subnet: {
            id: virtualnetwork.id
          }
        }
      }
    ]
  }
}


// I explicitly make this declaration of the vnet dependent on the VM's network interface so that there are no other vnet operations happening, and it doesn't try to happen before the NIC's IP is available
resource dnsUpdate 'Microsoft.Network/virtualNetworks@2019-11-01' = {
  name: vnetName
  dependsOn: [
    networkInterface
  ]
  location: resourceGroup().location
  properties: {
    dhcpOptions: {
       dnsServers: [
        networkInterface.properties.ipConfigurations[0].properties.privateIPAddress
       ]
    }
    addressSpace: {
      addressPrefixes: [
        vnetCIDR
      ]
    }
    subnets: [
      {
        name: subnetName
        properties: {
          networkSecurityGroup: {
            id: nsg.id          
          }
          addressPrefix: subnetCIDR
        }
      }
    ]
  }
}

[brwilkinson]

This is the solution that I have found to be 100% reliable for deploying 2 domain controllers in a new vnet.

I know you mentioned only 1 DC, so yours should be easier, however it may give you some ideas.

This is all just using Vnet DNS, nothing on any NICS.

https://github.com/brwilkinson/AzureDeploymentFramework/blob/60f2a6721034bacc441ad42a6c4701afaffe99f4/ADF/bicep/01-dp-rg-ALLRG.bicep#L486

Set vnet dns to

              "[variables('DC1PrivateIPAddress')]",
              "[variables('AzureDNS')]"
 
Deploy DC1

Set vnet dns to

              "[variables('DC1PrivateIPAddress')]",
 
Deploy DC2

Set vnet dns to

              "[variables('DC1PrivateIPAddress')]",
              "[variables('DC2PrivateIPAddress')]"
 
Which is the end state.

If you are using DSC to build the domain controllers, you likely also want to clear out and static IP that might get set inside the OS.

The task below in DSC (at the very end) then also clears local DNS addresses that get set on the servers and reboots, via a scheduled task.

Sample DSC configs
 
Primary: https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/ext-DSC/DSC-ADPrimary.ps1

Secondaries: https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/ext-DSC/DSC-ADSecondary.ps1

[brwilkinson]

I actually don't set the IP on the DC as static because per the documentation it should never really change underneath me, even if the VM is deallocated. From the vnet network interface docs:

The DCPROMO itself will set that static IP on the DC Operating System... so really you just want to ensure you undo that, if it happens.

--

Yeah unfortunately, in order for the Standalone Server to be able to download artifacts to install Extensions etc, such as the DSC extension, it requires a functioning DNS server to be set on the VNET... Azure DNS often makes the most sense for that initial state.

Once the DCPromo runs, the service itself is now functioning as a DNS Server (which is wasn't before) .. So now you will want to set the DNS on the VNET to only that single Domain controllers (totally remove the Azure DNS).

So since they are the 2 alternate states that are required, you do need to Orchestrate those states as part of the overall deployment of 1 or more Domain Controllers.

--

I still think I maybe missing something from your question however in relation to tying the NIC to the DNS Server ? Can you expand on this some more ?

By the way it is supported to set a NIC to have it's own Different DNS Server Configuration than the Subnet, so maybe that is the solution that you are looking for ? (It's not something that I have experimented with in relation to DC deployment, however there maybe a good scenario where it may work for you, like I said I haven't spent time thinking it over, since I am really happy with the orchestration of the VNET DNS).

[brwilkinson]

Sorry just to add, by not setting static IP addresses you cannot pre-determine what your Domain Controller IP Addresses will be, which you will need to know ahead of time or orchestrate setting the Primary DNS on the network.

I would also set them to something towards the end of the subnet space, to ensure no other servers takes the IP, prior to deploying the DC.

[jsblake]

I guess it would be clearer if I could or had posted my entire bicep file, but I have four steps:

1. Create vnet, don't specify any DNS settings. No depends on setting
2. Create domain controller, depends on step 1
3. Another definition of the vnet, this time with DNS setting referring to the IP address property of the domain controller's NIC. I set this explicitly to depend on step 2
4. Create the member server VMs, this is a module and I set it explicitly to depend on step 2

I feel like I'm using too many depends on statements and it's chaining this into a script instead of a declarative template that just describes my desired end state, but that may just be the reality I have to live with

[brwilkinson]

@jsblake It seems like you are doing everything right 🙂

Especially if you have a working reliable deployment. One thing about Bicep/ARM is that it does allow for this orchestration, which is required in bootstrapping environments sometimes. So like you say, it's not possible to simply define the 'end state' in those cases, since that is not enough.

Considering 1 and 3 are basically the same, you could use a module for those that take parameters with the DNS settings, then at least you are building consistency and code re-use. Especially once your VNET/Subnet starts to become more complex e.g. nsgs, delegations, rt's. So pass in the subnet info and the DNS info to the module, might be one enhancement.

[jsblake]

Oh, it is a module, I just didn't include that detail there. Anyway, thanks for the feedback and keep up the great work. Bicep has finally gotten us off some old rickety powershell scripts for our environment provisioning :)