Installing worker agent on Multiple CMF machines?

[RaiaN]

Describe the problem

Hi there,
We are trying to figure out how to quickly setup Deadline Cloud worker agent on multiple CMF workers?
We've tried to create a Windows OS template and clone it to all workers but then Worker agent ID is still the same.

What is the best approach?

Proposed Solution

Guidance on how to install Worker agent on multiple CMF machines

Example Use Cases

We are trying to figure out how to quickly setup Deadline Cloud worker agent on multiple CMF workers?
We've tried to create a Windows OS template and clone it to all workers but then Worker agent ID is still the same.

[RaiaN]

I figured the workaround for now:
worker_bootstrap = bootstrap_worker(config=config, use_existing_worker=False)

That will generate a new worker ID at startup

[jusiskin]

Hi @RaiaN, thanks for reaching out and reporting this issue.

This is a known issue. The worker agent stores the worker ID in the worker state file to use the same Deadline Cloud worker between worker agent launches.

If you make a machine image that includes this worker state file, then multiple worker hosts will try using the same worker and lead to problems. This is documented in two places:

1. Deadline Cloud developer guide → CMF → Create an Amazon Machine Image → "Prepare the Amazon EC2 instance" section

   • Step 8 instructs deleting the worker agent state.

2. The worker agent GitHub repo "Persistence Directory" section where it says:

   It is recommended if you are preparing an Amazon Machine Image (AMI) or a disk image, to delete the worker persistence directory before making the image in order to prevent all hosts using that image from running with the same worker ID.

An enhancement to the worker agent was recently merged (#527) that helps mitigate this problem and need for manual steps for CMFs on EC2. This change has not been released yet, but we will update when it has been. Note that this will only fix the issue if you are using a CMF on EC2. If you using workers outside of EC2, this problem remains open and you will need to ensure that you delete the worker agent state/persistence before making machine images.

Let me know if that makes sense. Unless you have any objections, I will resolve this issue once #527 has been released.

[RaiaN]

Ah, sorry, I guess I missed that PR!
Thanks again!

[RaiaN]

Hey, can we expect a cmdline option or like a new option added to worker.toml to FORCE regenerate worker ID at startup?

[jusiskin]

Hey, can we expect a cmdline option or like a new option added to worker.toml to FORCE regenerate worker ID at startup?

Hey @RaiaN,

I'd like to dig a bit further about your use-case for forcing generation of a new worker ID on startup. Is this required because you are you running non-EC2 workers and using machine images to setup the worker hosts?

[RaiaN]

Yes, exactly! When we work with non-EC2 workers we need to prepare an image. And each time we need to kill worker.json. This is error-prone.

[jusiskin]

Thanks for the context.

For on-premise workers, we would not want the worker agent to create a new worker ID every time the program starts for a few reasons:

1. If the worker host is restarted or the worker agent crashes, it would create a new worker. These workers would accumulate in the fleet and count towards the max worker total. They would need to be deleted to make room for workers to come online.
2. It would make it difficult to monitor job failures that are specific to a worker host if the worker changes each time the agent starts.

Workers in Deadline Cloud are intended to model a worker host machine and the worker [ID] should remain stable so long as the worker agent is running on the same host. What we need is a way for the agent to fingerprint the host machine and store the fingerprint in the worker.json. The worker agent does this already for EC2 hosts (see https://github.com/aws-deadline/deadline-cloud-worker-agent/blob/mainline/docs/state.md#how-the-state-file-affects-startup-behavior).

Some possible ways of fingerprinting an on-premise worker host:

Identifier	Comments
Hostname	This can change based based on IT needs
IP addresses	This can also change (e.g. DHCP)
MAC address(es)	Usually stable, but some systems randomize this to avoid fingerprinting
OS generated machine IDs (/etc/machine-id on Linux or HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\InstallationID on Windows)	Should investigate whether software for cloning machines from images reset this

This will require some investigation to see what the best approach is before we go to implement.

At the present, we are unable to prioritize this enhancement, but we are tracking this issue in our backlog and will provide updates here.

In the meantime, this issue can be worked around by deleting the worker.json file before creating a machine image, or after imaging each new machine.