Opennebula Tart driver fails to pull image from registry with a 404 error

Opennebula VM launched by Coin fails to boot VM and LCM State is BOOT_FAILURE.

Opennebula Log from https://one-master.ci.qt.io/#vms-tab/4372966 

Fri Mar 14 08:30:49 2025 [Z0][VM][I]: New state is ACTIVE
Fri Mar 14 08:30:49 2025 [Z0][VM][I]: New LCM state is PROLOG
Fri Mar 14 08:30:50 2025 [Z0][VM][I]: New LCM state is BOOT
Fri Mar 14 08:30:50 2025 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/4372966/deployment.0
Fri Mar 14 08:30:50 2025 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: ExitCode: 0
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: ExitCode: 0
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/mkdir -p.
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: ExitCode: 0
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/0/4372966/vm.xml.
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: ExitCode: 0
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: Successfully execute virtualization driver operation: /bin/cat - >/var/lib/one//datastores/0/4372966/ds.xml.
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: Command execution fail (exit code: 1): cat << 'EOT' | /var/tmp/one/vmm/tart/deploy '/var/lib/one//datastores/0/4372966/deployment.0' 'qt-srv-arm-mac39.on1.qt.io' 4372966 qt-srv-arm-mac39.on1.qt.io
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: ExitCode: 1
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: ExitCode: 0
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Fri Mar 14 08:30:51 2025 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Fri Mar 14 08:30:51 2025 [Z0][VMM][E]: DEPLOY: ExitCode: 1
Fri Mar 14 08:30:51 2025 [Z0][VM][I]: New LCM state is BOOT_FAILURE

deploy.log from tart host

qt-srv-arm-mac39:~ oneadmin$ cat datastores/0/4372966/deploy.log

2025-03-14 08:30:51,104 DEBUG Running /Users/oneadmin/tart.app/Contents/MacOS/tart clone --insecure tart-registry.on1.qt.io:5000/fefc3622175ae234fd88333cd5a918c8:CI one-4372966

2025-03-14 08:30:51,259 INFO stdout: b'pulling manifest...\n'

2025-03-14 08:30:51,259 ERROR stderr: b'Error: UnexpectedHTTPStatusCode(when: "pulling manifest", code: 404, details: "{\\"errors\\":[{\\"code\\":\\"MANIFEST_UNKNOWN\\",\\"message\\":\\"manifest unknown\\",\\"detail\\":{\\"Tag\\":\\"CI\\"}}]}\\n")\n'

2025-03-14 08:30:51,259 ERROR Failed to create virtual machine

Traceback (most recent call last):

  File "/var/tmp/one/vmm/tart/deploy", line 139, in main

    create_vm(vm_name, vm_id, xml)

  File "/var/tmp/one/vmm/tart/deploy", line 72, in create_vm

    tart("clone", "--insecure", "%s/%s:%s" % (REGISTRY, img, TAG), vm_name)

  File "/var/tmp/one/vmm/tart/deploy", line 42, in tart

    raise ConfigurationException("Command %s failed" % ' '.join(arguments))

ConfigurationException: Command /Users/oneadmin/tart.app/Contents/MacOS/tart clone --insecure tart-registry.on1.qt.io:5000/fefc3622175ae234fd88333cd5a918c8:CI one-4372966 failed

Running same tart pull command manually on the host does find the image and successfully pulls it.

This was a provisioning jobs, possible timing issue with image being pushed to registry and verification job attempting to pull it before registry was ready?