fix(megatron_training_lib): correct training-log parsing and drop sudo from libbnxt cp

[atnair-amd]

Summary

Three fixes in MegatronLlamaTrainingJob, all in
cvs/lib/megatron_training_lib.py. The first two share a single failure
mode — the parser silently sees the wrong slice of training.log and
downstream code treats an empty / mid-run metrics dict as PASS rather
than an explicit error. The third is a one-line drive-by in the same
file's container-exec path; included here rather than split out because
it's a single character change with no shared surface area.

What changed

1. TRAINING_PROGRESS_PATTERNS — match the final iter, not every iter

cvs/lib/megatron_training_lib.py (lines 44-46).

The previous patterns matched any throughput per GPU: / tokens/GPU/s
line. Megatron emits those every iteration after the first, so
_is_training_complete (call site at line 705) returned True after iter 1
and the poll loop tail-extracted metrics mid-run.

Replaced with a single pattern anchored on megatron's final-iter marker:

TRAINING_PROGRESS_PATTERNS = [
    r'iteration\s+(\d+)/\s+\1\s*\|',
]

The \1 backreference requires current-iter == total-iter, so the
pattern fires exactly once per run (e.g. iteration 100/ 100 | ...).

2. get_training_results_dict — filter before tail

cvs/lib/megatron_training_lib.py (~line 587).

Was:

out_dict = self.phdl.exec(
    f'cat {self.log_dir}/megatron-logs/out-node{last_node_num}/training.log | tail -15'
)

On a healthy run, post-training cleanup (aiter import spam, validation eval
banner, NCCL teardown warnings) writes well over 15 lines after the final
iter, pushing every iteration N/... line out of the tail -15 window.
_parse_training_results then matches nothing and returns an empty
results_dict, which downstream code treats as PASS.

Now:

out_dict = self.phdl.exec(
    f"grep -E 'iteration[[:space:]]+[0-9]+/[[:space:]]+[0-9]+[[:space:]]*\\|' "
    f"{self.log_dir}/megatron-logs/out-node{last_node_num}/training.log | tail -15"
)

The grep keeps only iteration lines; tail -15 then yields the last 15
of those, which is what _parse_training_results was already assumed to
be fed.

3. exec_nic_setup_scripts — drop sudo from the libbnxt docker exec

cvs/lib/megatron_training_lib.py (~line 387, broadcom/thor branch).

The training container already runs as root, so the nested sudo in:

f'docker exec {self.container_name} /bin/bash -c "sudo cp ... so.host ... so; ..."'

is unnecessary. On training images that don't install sudo at all, the
exec returns 127 and the libbnxt copy never happens — surfacing later in
the run as RDMA / verbs failures rather than a clean setup error.
Dropping the sudo prefix makes the command work on both flavors of
image; root-in-container can cp into /usr/lib directly.

[anujmittal-amd]

r'iteration\s+(\d+)/\s+\1\s*|' requires whitespace after /. If Megatron logs ever emit iteration 100/100 | or vary spacing by version, _is_training_complete() may stop matching entirely.
Would r'iteration\s+(\d+)\s*/\s*\1\s*|' be safer here?

[atnair-amd]

Good catch — switched to r'iteration\s+(\d+)\s*/\s*\1\s*\|' in f91e6c8 so spacing around / is tolerated.

[anujmittal-amd]

Same comment as above

[atnair-amd]

Same fix applied to the grep ERE in f91e6c8: [[:space:]]+/[[:space:]]+ → [[:space:]]*/[[:space:]]* around the /.