Fix submission_checker import: robust fallback with importlib and sys.path priority

[Copilot]

The import submission_checker as checker fallback path in generate-mlperf-inference-user-conf/customize.py was not wrapped in a try/except block, and the sys.path.append call placed the automotive tools/submission directory at the end of sys.path — causing the wrong (or no) module to be found, and log_parser (an internal dependency of submission_checker.py) to be unfindable, resulting in a silently-caught ModuleNotFoundError and ultimately the AttributeError: submission_checker module is missing required constants for file checks CI failure on all non-Docker jobs.

Changes

• Both imports wrapped in try/except — import submission_checker.constants as constants and import submission_checker as checker are both wrapped in try/except (ImportError, ModuleNotFoundError, AttributeError).

• sys.path.insert(0, ...) instead of sys.path.append — the automotive/inference tools/submission directory is inserted at the front of sys.path so it takes precedence over any other copy of the module on sys.path, and log_parser.py (a dependency of submission_checker.py) is reliably found.

• importlib.util.spec_from_file_location fallback — when both standard imports fail (e.g., due to sys.modules caching or sys.path ordering), the module is loaded directly from the file, bypassing the Python import cache entirely. The directory is inserted at the front of sys.path before exec_module so log_parser is found.

• _get_submission_checker_dir(env) helper — eliminates duplicated os.path.join(env['MLC_MLPERF_INFERENCE_SOURCE'], "tools", "submission") path construction across get_checker_files() and get_required_min_queries_offline().

• import importlib.util moved to top-level imports and the importlib fallback's except clause narrowed to specific types (ImportError, ModuleNotFoundError, AttributeError, OSError).

def _get_submission_checker_constants_module(submission_checker_dir=None):
    required_attrs = ["REQUIRED_ACC_FILES", "REQUIRED_PERF_FILES", ...]

    if submission_checker_dir and submission_checker_dir not in sys.path:
        sys.path.insert(0, submission_checker_dir)

    try:
        import submission_checker.constants as constants
        if all(hasattr(constants, attr) for attr in required_attrs):
            return constants
    except (ImportError, ModuleNotFoundError, AttributeError):
        pass

    try:
        import submission_checker as checker
        if all(hasattr(checker, attr) for attr in required_attrs):
            return checker
    except (ImportError, ModuleNotFoundError, AttributeError):
        pass

    # importlib fallback: bypasses sys.modules caching and sys.path ordering
    if submission_checker_dir:
        checker_file = os.path.join(submission_checker_dir, "submission_checker.py")
        if os.path.isfile(checker_file):
            try:
                spec = importlib.util.spec_from_file_location(
                    "submission_checker_direct", checker_file)
                mod = importlib.util.module_from_spec(spec)
                spec.loader.exec_module(mod)
                if all(hasattr(mod, attr) for attr in required_attrs):
                    return mod
            except (ImportError, ModuleNotFoundError, AttributeError, OSError):
                pass

    raise AttributeError(
        "submission_checker module is missing required constants for file checks"
    )

🧾 PR Checklist

• Target branch is dev

📌 Note: PRs must be raised against dev. Do not commit directly to main.

📁 File Hygiene & Output Handling

• No unintended files (e.g., logs, cache, temp files, pycache, output folders) are committed

📝 Comments & Communication

• Proper inline comments are added to explain important or non-obvious changes
• PR title and description clearly state what the PR does and why

🛡️ Safety & Security

• No secrets or credentials are committed
• Paths, shell commands, and environment handling are safe and portable

Original prompt

The CI build for PR #940 is failing with:

AttributeError: submission_checker module is missing required constants for file checks
  at /home/runner/MLC/repos/GATEOverflow@mlperf-automations/script/run-mlperf-automotive-app/customize.py:265 in preprocess

The root cause is in script/generate-mlperf-inference-user-conf/customize.py in the _get_submission_checker_constants_module() function (around line 584-607).

The first import (import submission_checker.constants as constants) is correctly wrapped in a try/except block, but the second import (import submission_checker as checker on line 601) is NOT wrapped in a try/except block. If that import fails or the module doesn't have the required attributes, it either raises an unhandled ImportError or falls through to raise the AttributeError without a proper fallback.

Fix needed: Wrap the second import submission_checker as checker block (lines 601-603) in its own try/except (ImportError, ModuleNotFoundError, AttributeError) block, consistent with the pattern used for the first import above it.

The fixed code should look like:

def _get_submission_checker_constants_module():
    required_attrs = [
        "REQUIRED_ACC_FILES",
        "REQUIRED_PERF_FILES",
        "REQUIRED_POWER_FILES",
        "REQUIRED_PERF_POWER_FILES",
        "REQUIRED_MEASURE_FILES",
        "OFFLINE_MIN_SPQ_SINCE_V4",
    ]

    try:
        import submission_checker.constants as constants
        if all(hasattr(constants, attr) for attr in required_attrs):
            return constants
    except (ImportError, ModuleNotFoundError, AttributeError):
        pass

    try:
        import submission_checker as checker
        if all(hasattr(checker, attr) for attr in required_attrs):
            return checker
    except (ImportError, ModuleNotFoundError, AttributeError):
        pass

    raise AttributeError(
        "submission_checker module is missing required constants for file checks"
    )

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: How can I fix this build?

Assistant: The build is failing with this error:

AttributeError: submission_checker module is missing required constants for file checks

This occurs at script/run-mlperf-automotive-app/customize.py:265 in the preprocess function.

Let me look at that file for more context:

Assistant: ## Build Failure Analysis

The job is failing with this error:

AttributeError: submission_checker module is missing required constants for file checks

This occurs at script/run-mlperf-automotive-app/customize.py:265 during the preprocess phase, when the app,abtf-inference script is invoked.

Root Cause

The script depends on the submission_checker module from the MLPerf automotive benchmark source (cloned into get-git-repo_automotive-src). That module is expected to expose certain constants for file checks, but the version being pulled no longer has them (or they've been renamed/restructured).

How to Fix

You have a few options:

1. Pin the automotive source to a compatible commit — If the get,src,source,automotive dependency pulls the latest main branch, the upstream repo may have changed. Add a specific commit or tag to the dependency to ensure compatibility.

2. Update the code to match the new submission_checker API — Check what constants the automotive source's submission_checker.py now exports and update the consuming code accordingly. The error likely originates in a script called by mlc.access(ii) (the app,abtf-inference implementation script) that imports submission_checker and references constants that no longer exist.

3. Check PR Support bert and resnet run using Nvidia6.0 code #940 changes — If this PR modified dependencies or version references for the automotive source, verify that the referenced version of the automotive repo is compatible with the checker logic used in the automation scripts.

Would you like me to look at any specific file (e.g., the script that actually imports and validates against submission_checker) for more detail?

User: The PR is having changes to ‎script/generate-mlperf-inference-user-conf/customize.py and i see the try and catch block being handled wrongly. Can you fix it?

Assistant: Now I can see the issue. Let me look at the current state of the file on the PR branch:

Assistant: Now I can see the issue. Let me look at the current file on the PR branch to get the exact content:

Assistant: Now I can see the issue. In _get_submission_checker_constants_module() (lines 584-607), the second import submission_checker as checker on line 601 is not wrapped in a try/except block. ...

This pull request was created from Copilot chat.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[arjunsuresh]

@copilot
fix this
https://github.com/mlcommons/mlperf-automations/actions/runs/25408961644/job/74526129014?pr=941#step:11:1457

[Copilot]

@copilot
fix this
https://github.com/mlcommons/mlperf-automations/actions/runs/25408961644/job/74526129014?pr=941#step:11:1457

Fixed in commit 70aaf4c. The root cause was that import submission_checker as checker was silently failing (caught by my try/except) because:

1. sys.path.append(...) placed the automotive tools/submission dir at the end of sys.path — if any earlier entry provided a conflicting or empty submission_checker, the automotive one was never reached.
2. This caused from log_parser import MLPerfLog inside submission_checker.py to raise ModuleNotFoundError, which was swallowed.

The fix:

• Changed sys.path.append → sys.path.insert(0, submission_checker_dir) so the automotive checker directory always takes precedence.
• Added an importlib.util.spec_from_file_location fallback that loads submission_checker.py directly from the file, completely bypassing sys.modules caching and sys.path ordering.
• Extracted _get_submission_checker_dir(env) helper to avoid duplicating the path construction.
• Moved import importlib.util to the top-level imports and narrowed the except clause to specific exception types.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/repos/mlcommons/mlperf_automotive/contents/tools/submission
  • Triggering command: /usr/bin/curl curl -s REDACTED (http block)
• https://api.github.com/repos/mlcommons/mlperf_automotive/git/trees/ee526dc63d9ca2636000343c5d2d16132145719e
  • Triggering command: /usr/bin/curl curl -s REDACTED (http block)
  • Triggering command: /usr/bin/curl curl -s REDACTED time/copilot-developer-action-ma/home/REDACTED/work/mlperf-automations/mlperf-automations/script/get-mlperf-automotive-utils/customize.py conf�� get --global ndor/bin/bash user.email checker_constant-w k/mlperf-automat-t git conf�� get --local /usr/local/.ghcu168.63.129.16 credential.helpebash (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)