2024-05-08T06:36:24.474 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8
2024-05-08T06:36:24.475 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333 --owner scheduled_adking@teuthology --name adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi --description orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} -- /home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333/orig.config.yaml
2024-05-08T06:36:24.502 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-05-08T06:36:24.590 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333
  branch: wip-adk-testing-2024-05-07-0802
  description: orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop
    1-start 2-services/nfs2 3-final}
  email: adking@redhat.com
  first_in_suite: false
  job_id: '7697333'
  kernel:
    kdb: 1
    sha1: distro
  last_in_suite: false
  machine_type: smithi
  name: adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi
  no_nested_subset: false
  nuke-on-error: true
  openstack:
  - volumes:
      count: 4
      size: 10
  os_type: centos
  os_version: 9.stream
  overrides:
    admin_socket:
      branch: wip-adk-testing-2024-05-07-0802
    ceph:
      conf:
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
        osd:
          debug ms: 1
          debug osd: 20
          osd shutdown pgref assert: true
      flavor: default
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      log-only-match:
      - CEPHADM_
      sha1: 7c8f650b36e258f639fa4a83becade57cbfd2009
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon: {}
    install:
      ceph:
        flavor: default
        sha1: 7c8f650b36e258f639fa4a83becade57cbfd2009
    selinux:
      allowlist:
      - scontext=system_u:system_r:logrotate_t:s0
    workunit:
      branch: wip-adk-testing-2024-05-07-0802
      sha1: 7c8f650b36e258f639fa4a83becade57cbfd2009
  owner: scheduled_adking@teuthology
  priority: 80
  repo: https://git.ceph.com/ceph-ci.git
  roles:
  - - host.a
    - client.0
  - - host.b
    - client.1
  seed: 1064
  sha1: 7c8f650b36e258f639fa4a83becade57cbfd2009
  sleep_before_teardown: 0
  subset: 1/20
  suite: orch:cephadm
  suite_branch: wip-adk-testing-2024-05-07-0802
  suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa
  suite_relpath: qa
  suite_repo: https://git.ceph.com/ceph-ci.git
  suite_sha1: 7c8f650b36e258f639fa4a83becade57cbfd2009
  targets:
    smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ/wFSBqluEpbrthgBx8g+89AZVchcHlkbztzGTawEssYvQORjz4qIbOmrca9WYuvIENvE+RwLMjLcbdlPOkDZQ=
    smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9f/VAz71HOhMPSuDwGIz6HngNnhDL7Cvx7FQ9ZXc9MiDZWWr7IdL2IUPX2bwmxnHCs+sNADkziBblZ19QhZDI=
  tasks:
  - pexec:
      all:
      - sudo dnf install runc -y
      - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf
      - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf
  - nvme_loop: null
  - cephadm:
      roleless: true
  - cephadm.shell:
      host.a:
      - ceph orch status
      - ceph orch ps
      - ceph orch ls
      - ceph orch host ls
      - ceph orch device ls
  - vip.exec:
      all-hosts:
      - systemctl stop nfs-server
  - cephadm.shell:
      host.a:
      - ceph nfs cluster create foo
  - cephadm.wait_for_service:
      service: nfs.foo
  - cephadm.shell:
      host.a:
      - stat -c '%u %g' /var/log/ceph | grep '167 167'
      - ceph orch status
      - ceph orch ps
      - ceph orch ls
      - ceph orch host ls
      - ceph orch device ls
      - ceph orch ls | grep '^osd.all-available-devices '
  teuthology:
    fragments_dropped: []
    meta: {}
    postmerge: []
  teuthology_branch: main
  teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1
  timestamp: 2024-05-08_05:24:58
  tube: smithi
  user: adking
  verbose: false
  worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885
2024-05-08T06:36:24.590 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa; will attempt to use it
2024-05-08T06:36:24.590 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa/tasks
2024-05-08T06:36:24.591 INFO:teuthology.run_tasks:Running task internal.check_packages...
2024-05-08T06:36:24.592 INFO:teuthology.task.internal:Checking packages...
2024-05-08T06:36:24.620 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7c8f650b36e258f639fa4a83becade57cbfd2009'
2024-05-08T06:36:24.620 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2024-05-08T06:36:24.620 INFO:teuthology.packaging:ref: None
2024-05-08T06:36:24.620 INFO:teuthology.packaging:tag: None
2024-05-08T06:36:24.620 INFO:teuthology.packaging:branch: wip-adk-testing-2024-05-07-0802
2024-05-08T06:36:24.620 INFO:teuthology.packaging:sha1: 7c8f650b36e258f639fa4a83becade57cbfd2009
2024-05-08T06:36:24.620 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-05-07-0802
2024-05-08T06:36:24.839 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3593.g7c8f650b
2024-05-08T06:36:24.841 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2024-05-08T06:36:24.847 INFO:teuthology.task.internal:no buildpackages task found
2024-05-08T06:36:24.847 INFO:teuthology.run_tasks:Running task internal.save_config...
2024-05-08T06:36:24.861 INFO:teuthology.task.internal:Saving configuration
2024-05-08T06:36:24.871 INFO:teuthology.run_tasks:Running task internal.check_lock...
2024-05-08T06:36:24.877 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-05-08T06:36:24.902 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 06:31:10.567177', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ/wFSBqluEpbrthgBx8g+89AZVchcHlkbztzGTawEssYvQORjz4qIbOmrca9WYuvIENvE+RwLMjLcbdlPOkDZQ='}
2024-05-08T06:36:24.925 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 06:31:10.569182', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9f/VAz71HOhMPSuDwGIz6HngNnhDL7Cvx7FQ9ZXc9MiDZWWr7IdL2IUPX2bwmxnHCs+sNADkziBblZ19QhZDI='}
2024-05-08T06:36:24.925 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2024-05-08T06:36:24.932 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['host.a', 'client.0']
2024-05-08T06:36:24.932 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['host.b', 'client.1']
2024-05-08T06:36:24.932 INFO:teuthology.run_tasks:Running task console_log...
2024-05-08T06:36:25.007 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f801b291790>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f801c07d160>, signals=[15])
2024-05-08T06:36:25.007 INFO:teuthology.run_tasks:Running task internal.connect...
2024-05-08T06:36:25.014 INFO:teuthology.task.internal:Opening connections...
2024-05-08T06:36:25.014 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com
2024-05-08T06:36:25.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:36:25.092 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com
2024-05-08T06:36:25.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:36:25.167 INFO:teuthology.run_tasks:Running task internal.push_inventory...
2024-05-08T06:36:25.174 DEBUG:teuthology.orchestra.run.smithi031:> uname -m
2024-05-08T06:36:25.192 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64
2024-05-08T06:36:25.193 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release
2024-05-08T06:36:25.250 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream"
2024-05-08T06:36:25.250 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9"
2024-05-08T06:36:25.250 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos"
2024-05-08T06:36:25.250 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora"
2024-05-08T06:36:25.250 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9"
2024-05-08T06:36:25.250 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9"
2024-05-08T06:36:25.250 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9"
2024-05-08T06:36:25.250 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31"
2024-05-08T06:36:25.251 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon"
2024-05-08T06:36:25.251 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-05-08T06:36:25.251 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/"
2024-05-08T06:36:25.251 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-05-08T06:36:25.251 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-05-08T06:36:25.251 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-05-08T06:36:25.251 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server
2024-05-08T06:36:25.276 DEBUG:teuthology.orchestra.run.smithi083:> uname -m
2024-05-08T06:36:25.294 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64
2024-05-08T06:36:25.294 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release
2024-05-08T06:36:25.351 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream"
2024-05-08T06:36:25.351 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9"
2024-05-08T06:36:25.351 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos"
2024-05-08T06:36:25.351 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora"
2024-05-08T06:36:25.351 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9"
2024-05-08T06:36:25.352 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9"
2024-05-08T06:36:25.352 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9"
2024-05-08T06:36:25.352 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31"
2024-05-08T06:36:25.352 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon"
2024-05-08T06:36:25.352 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-05-08T06:36:25.352 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/"
2024-05-08T06:36:25.352 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-05-08T06:36:25.352 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-05-08T06:36:25.352 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-05-08T06:36:25.352 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server
2024-05-08T06:36:25.376 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles...
2024-05-08T06:36:25.384 INFO:teuthology.run_tasks:Running task internal.check_conflict...
2024-05-08T06:36:25.396 INFO:teuthology.task.internal:Checking for old test directory...
2024-05-08T06:36:25.396 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest
2024-05-08T06:36:25.398 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest
2024-05-08T06:36:25.416 INFO:teuthology.run_tasks:Running task internal.check_ceph_data...
2024-05-08T06:36:25.422 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph...
2024-05-08T06:36:25.422 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph)
2024-05-08T06:36:25.458 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph)
2024-05-08T06:36:25.536 INFO:teuthology.run_tasks:Running task internal.vm_setup...
2024-05-08T06:36:25.723 INFO:teuthology.run_tasks:Running task kernel...
2024-05-08T06:36:25.737 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'}
2024-05-08T06:36:25.737 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300
2024-05-08T06:36:25.737 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv
2024-05-08T06:36:25.738 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv
2024-05-08T06:36:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:36:25.755 DEBUG:teuthology.orchestra.run.smithi083:> uname -r
2024-05-08T06:36:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:36:25.756 DEBUG:teuthology.orchestra.run.smithi031:> uname -r
2024-05-08T06:36:25.812 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-437.el9.x86_64
2024-05-08T06:36:25.812 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-437.el9.x86_64
2024-05-08T06:36:25.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel
2024-05-08T06:36:25.814 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-437.el9.x86_64
2024-05-08T06:36:25.814 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-437.el9.x86_64
2024-05-08T06:36:25.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel
2024-05-08T06:36:28.255 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS                         11 MB/s | 8.1 MB     00:00
2024-05-08T06:36:30.006 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream                      29 MB/s |  19 MB     00:00
2024-05-08T06:36:30.051 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS                        3.0 MB/s | 8.1 MB     00:02
2024-05-08T06:36:31.802 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream                      31 MB/s |  19 MB     00:00
2024-05-08T06:36:35.682 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB                           3.1 MB/s | 6.2 MB     00:02
2024-05-08T06:36:37.199 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages                95 kB/s |  16 kB     00:00
2024-05-08T06:36:37.928 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB                           2.6 MB/s | 6.2 MB     00:02
2024-05-08T06:36:38.909 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux              13 MB/s |  21 MB     00:01
2024-05-08T06:36:39.574 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages                38 kB/s |  16 kB     00:00
2024-05-08T06:36:41.480 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux              12 MB/s |  21 MB     00:01
2024-05-08T06:36:44.067 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras                                       15 kB/s | 1.7 kB     00:00
2024-05-08T06:36:46.048 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed.
2024-05-08T06:36:46.049 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed.
2024-05-08T06:36:46.139 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved.
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout: Package                   Architecture Version              Repository    Size
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout:Installing:
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout: kernel                    x86_64       5.14.0-444.el9       baseos       484 k
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies:
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core               x86_64       5.14.0-444.el9       baseos        15 M
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules            x86_64       5.14.0-444.el9       baseos        34 M
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core       x86_64       5.14.0-444.el9       baseos        28 M
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:36:46.145 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary
2024-05-08T06:36:46.146 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================
2024-05-08T06:36:46.146 INFO:teuthology.orchestra.run.smithi083.stdout:Install  4 Packages
2024-05-08T06:36:46.146 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:36:46.146 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 78 M
2024-05-08T06:36:46.146 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 123 M
2024-05-08T06:36:46.146 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages:
2024-05-08T06:36:46.485 INFO:teuthology.orchestra.run.smithi083.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm         2.1 MB/s | 484 kB     00:00
2024-05-08T06:36:46.763 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras                                       46 kB/s | 1.7 kB     00:00
2024-05-08T06:36:47.002 INFO:teuthology.orchestra.run.smithi083.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm     21 MB/s |  15 MB     00:00
2024-05-08T06:36:47.303 INFO:teuthology.orchestra.run.smithi083.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm  33 MB/s |  34 MB     00:01
2024-05-08T06:36:47.670 INFO:teuthology.orchestra.run.smithi083.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6  24 MB/s |  28 MB     00:01
2024-05-08T06:36:47.670 INFO:teuthology.orchestra.run.smithi083.stdout:--------------------------------------------------------------------------------
2024-05-08T06:36:47.670 INFO:teuthology.orchestra.run.smithi083.stdout:Total                                            51 MB/s |  78 MB     00:01
2024-05-08T06:36:48.350 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check
2024-05-08T06:36:48.695 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded.
2024-05-08T06:36:48.695 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test
2024-05-08T06:36:48.827 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed.
2024-05-08T06:36:48.827 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed.
2024-05-08T06:36:48.913 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved.
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout: Package                   Architecture Version              Repository    Size
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout:Installing:
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout: kernel                    x86_64       5.14.0-444.el9       baseos       484 k
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies:
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core               x86_64       5.14.0-444.el9       baseos        15 M
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules            x86_64       5.14.0-444.el9       baseos        34 M
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core       x86_64       5.14.0-444.el9       baseos        28 M
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================
2024-05-08T06:36:48.924 INFO:teuthology.orchestra.run.smithi031.stdout:Install  4 Packages
2024-05-08T06:36:48.925 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:36:48.925 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 78 M
2024-05-08T06:36:48.925 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 123 M
2024-05-08T06:36:48.925 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages:
2024-05-08T06:36:49.616 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded.
2024-05-08T06:36:49.616 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction
2024-05-08T06:36:49.964 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm         736 kB/s | 484 kB     00:00
2024-05-08T06:36:51.998 INFO:teuthology.orchestra.run.smithi083.stdout:  Preparing        :                                                        1/1
2024-05-08T06:36:52.877 INFO:teuthology.orchestra.run.smithi083.stdout:  Installing       : kernel-modules-core-5.14.0-444.el9.x86_64              1/4
2024-05-08T06:36:52.962 INFO:teuthology.orchestra.run.smithi083.stdout:  Installing       : kernel-core-5.14.0-444.el9.x86_64                      2/4
2024-05-08T06:36:55.607 INFO:teuthology.orchestra.run.smithi083.stdout:  Running scriptlet: kernel-core-5.14.0-444.el9.x86_64                      2/4
2024-05-08T06:36:55.735 INFO:teuthology.orchestra.run.smithi083.stdout:  Installing       : kernel-modules-5.14.0-444.el9.x86_64                   3/4
2024-05-08T06:37:00.422 INFO:teuthology.orchestra.run.smithi083.stdout:  Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64                   3/4
2024-05-08T06:37:00.529 INFO:teuthology.orchestra.run.smithi083.stdout:  Installing       : kernel-5.14.0-444.el9.x86_64                           4/4
2024-05-08T06:37:05.266 INFO:teuthology.orchestra.run.smithi083.stdout:  Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64              4/4
2024-05-08T06:37:09.612 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm    778 kB/s |  15 MB     00:20
2024-05-08T06:37:27.651 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): kernel-modules-core-5.14.0-444.el9.x86_6 760 kB/s |  28 MB     00:37
2024-05-08T06:37:29.126 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 872 kB/s |  34 MB     00:39
2024-05-08T06:37:29.126 INFO:teuthology.orchestra.run.smithi031.stdout:--------------------------------------------------------------------------------
2024-05-08T06:37:29.126 INFO:teuthology.orchestra.run.smithi031.stdout:Total                                           1.9 MB/s |  78 MB     00:40
2024-05-08T06:37:29.835 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check
2024-05-08T06:37:30.248 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded.
2024-05-08T06:37:30.249 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test
2024-05-08T06:37:31.218 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded.
2024-05-08T06:37:31.218 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction
2024-05-08T06:37:33.757 INFO:teuthology.orchestra.run.smithi031.stdout:  Preparing        :                                                        1/1
2024-05-08T06:37:34.416 INFO:teuthology.orchestra.run.smithi031.stdout:  Installing       : kernel-modules-core-5.14.0-444.el9.x86_64              1/4
2024-05-08T06:37:34.500 INFO:teuthology.orchestra.run.smithi031.stdout:  Installing       : kernel-core-5.14.0-444.el9.x86_64                      2/4
2024-05-08T06:37:34.869 INFO:teuthology.orchestra.run.smithi083.stdout:  Running scriptlet: kernel-core-5.14.0-444.el9.x86_64                      4/4
2024-05-08T06:37:34.907 INFO:teuthology.orchestra.run.smithi083.stdout:  Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64                   4/4
2024-05-08T06:37:35.215 INFO:teuthology.orchestra.run.smithi083.stdout:  Running scriptlet: kernel-5.14.0-444.el9.x86_64                           4/4
2024-05-08T06:37:35.215 INFO:teuthology.orchestra.run.smithi083.stdout:  Verifying        : kernel-5.14.0-444.el9.x86_64                           1/4
2024-05-08T06:37:35.215 INFO:teuthology.orchestra.run.smithi083.stdout:  Verifying        : kernel-core-5.14.0-444.el9.x86_64                      2/4
2024-05-08T06:37:35.215 INFO:teuthology.orchestra.run.smithi083.stdout:  Verifying        : kernel-modules-5.14.0-444.el9.x86_64                   3/4
2024-05-08T06:37:36.020 INFO:teuthology.orchestra.run.smithi083.stdout:  Verifying        : kernel-modules-core-5.14.0-444.el9.x86_64              4/4
2024-05-08T06:37:36.020 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:37:36.020 INFO:teuthology.orchestra.run.smithi083.stdout:Installed:
2024-05-08T06:37:36.020 INFO:teuthology.orchestra.run.smithi083.stdout:  kernel-5.14.0-444.el9.x86_64
2024-05-08T06:37:36.020 INFO:teuthology.orchestra.run.smithi083.stdout:  kernel-core-5.14.0-444.el9.x86_64
2024-05-08T06:37:36.020 INFO:teuthology.orchestra.run.smithi083.stdout:  kernel-modules-5.14.0-444.el9.x86_64
2024-05-08T06:37:36.020 INFO:teuthology.orchestra.run.smithi083.stdout:  kernel-modules-core-5.14.0-444.el9.x86_64
2024-05-08T06:37:36.020 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:37:36.020 INFO:teuthology.orchestra.run.smithi083.stdout:Complete!
2024-05-08T06:37:36.395 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1
2024-05-08T06:37:36.458 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-444.el9.x86_64
2024-05-08T06:37:36.458 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64
2024-05-08T06:37:36.458 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64
2024-05-08T06:37:36.458 INFO:teuthology.task.kernel:Skipping firmware on distro kernel
2024-05-08T06:37:36.458 DEBUG:teuthology.task.kernel:src is distro, skipping download
2024-05-08T06:37:36.459 INFO:teuthology.task.kernel:Installing distro kernel on host.b...
2024-05-08T06:37:36.459 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro)
2024-05-08T06:37:36.459 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1
2024-05-08T06:37:36.507 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-444.el9.x86_64
2024-05-08T06:37:36.507 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64
2024-05-08T06:37:36.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Name        : grub2-tools
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch       : 1
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Version     : 2.06
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Release     : 68.el9
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Group       : Unspecified
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Size        : 8264254
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:License     : GPLv3+
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Signature   : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM  : grub2-2.06-68.el9.src.rpm
2024-05-08T06:37:36.548 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date  : Wed 02 Aug 2023 03:15:48 PM UTC
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host  : x86-05.stream.rdu2.redhat.com
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:Packager    : builder@centos.org
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor      : CentOS
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:URL         : http://www.gnu.org/software/grub/
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:Summary     : Support tools for GRUB.
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:Description :
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture.  It supports a rich
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices.
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:37:36.549 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms.
2024-05-08T06:37:36.550 INFO:teuthology.task.kernel:Updating Grub Version: grub2
2024-05-08T06:37:36.550 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 5.14.0-444.el9.x86_64
2024-05-08T06:37:36.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg
2024-05-08T06:37:37.152 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ...
2024-05-08T06:37:37.307 INFO:teuthology.orchestra.run.smithi031.stdout:  Running scriptlet: kernel-core-5.14.0-444.el9.x86_64                      2/4
2024-05-08T06:37:37.440 INFO:teuthology.orchestra.run.smithi031.stdout:  Installing       : kernel-modules-5.14.0-444.el9.x86_64                   3/4
2024-05-08T06:37:38.463 INFO:teuthology.orchestra.run.smithi083.stderr:Adding boot menu entry for UEFI Firmware Settings ...
2024-05-08T06:37:38.495 INFO:teuthology.orchestra.run.smithi083.stderr:done
2024-05-08T06:37:38.497 DEBUG:teuthology.orchestra.run.smithi083:> mktemp
2024-05-08T06:37:38.512 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.gs5anPIpNV
2024-05-08T06:37:38.513 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gs5anPIpNV
2024-05-08T06:37:38.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.gs5anPIpNV
2024-05-08T06:37:38.712 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.gs5anPIpNV is 6KB
2024-05-08T06:37:38.761 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.gs5anPIpNV
2024-05-08T06:37:38.775 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true
2024-05-08T06:37:38.836 INFO:teuthology.orchestra.run.smithi083.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf
2024-05-08T06:37:38.836 INFO:teuthology.orchestra.run.smithi083.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf
2024-05-08T06:37:38.836 INFO:teuthology.orchestra.run.smithi083.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf
2024-05-08T06:37:38.837 INFO:teuthology.orchestra.run.smithi083.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf
2024-05-08T06:37:38.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64
2024-05-08T06:37:38.969 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now
2024-05-08T06:37:42.525 INFO:teuthology.orchestra.run.smithi031.stdout:  Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64                   3/4
2024-05-08T06:37:42.576 INFO:teuthology.orchestra.run.smithi031.stdout:  Installing       : kernel-5.14.0-444.el9.x86_64                           4/4
2024-05-08T06:37:47.667 INFO:teuthology.orchestra.run.smithi031.stdout:  Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64              4/4
2024-05-08T06:38:08.972 INFO:teuthology.task.kernel:Checking client host.b for new kernel version...
2024-05-08T06:38:08.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com'
2024-05-08T06:38:08.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:38:18.645 INFO:teuthology.orchestra.run.smithi031.stdout:  Running scriptlet: kernel-core-5.14.0-444.el9.x86_64                      4/4
2024-05-08T06:38:18.686 INFO:teuthology.orchestra.run.smithi031.stdout:  Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64                   4/4
2024-05-08T06:38:19.045 INFO:teuthology.orchestra.run.smithi031.stdout:  Running scriptlet: kernel-5.14.0-444.el9.x86_64                           4/4
2024-05-08T06:38:19.045 INFO:teuthology.orchestra.run.smithi031.stdout:  Verifying        : kernel-5.14.0-444.el9.x86_64                           1/4
2024-05-08T06:38:19.046 INFO:teuthology.orchestra.run.smithi031.stdout:  Verifying        : kernel-core-5.14.0-444.el9.x86_64                      2/4
2024-05-08T06:38:19.046 INFO:teuthology.orchestra.run.smithi031.stdout:  Verifying        : kernel-modules-5.14.0-444.el9.x86_64                   3/4
2024-05-08T06:38:19.836 INFO:teuthology.orchestra.run.smithi031.stdout:  Verifying        : kernel-modules-core-5.14.0-444.el9.x86_64              4/4
2024-05-08T06:38:19.836 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:38:19.836 INFO:teuthology.orchestra.run.smithi031.stdout:Installed:
2024-05-08T06:38:19.836 INFO:teuthology.orchestra.run.smithi031.stdout:  kernel-5.14.0-444.el9.x86_64
2024-05-08T06:38:19.836 INFO:teuthology.orchestra.run.smithi031.stdout:  kernel-core-5.14.0-444.el9.x86_64
2024-05-08T06:38:19.836 INFO:teuthology.orchestra.run.smithi031.stdout:  kernel-modules-5.14.0-444.el9.x86_64
2024-05-08T06:38:19.836 INFO:teuthology.orchestra.run.smithi031.stdout:  kernel-modules-core-5.14.0-444.el9.x86_64
2024-05-08T06:38:19.837 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:38:19.837 INFO:teuthology.orchestra.run.smithi031.stdout:Complete!
2024-05-08T06:38:20.181 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1
2024-05-08T06:38:20.253 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-444.el9.x86_64
2024-05-08T06:38:20.254 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64
2024-05-08T06:38:20.255 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64
2024-05-08T06:38:20.255 INFO:teuthology.task.kernel:Skipping firmware on distro kernel
2024-05-08T06:38:20.255 DEBUG:teuthology.task.kernel:src is distro, skipping download
2024-05-08T06:38:20.255 INFO:teuthology.task.kernel:Installing distro kernel on host.a...
2024-05-08T06:38:20.255 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro)
2024-05-08T06:38:20.255 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1
2024-05-08T06:38:20.305 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-444.el9.x86_64
2024-05-08T06:38:20.306 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64
2024-05-08T06:38:20.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:Name        : grub2-tools
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch       : 1
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:Version     : 2.06
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:Release     : 68.el9
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:Group       : Unspecified
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:Size        : 8264254
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:License     : GPLv3+
2024-05-08T06:38:20.341 INFO:teuthology.orchestra.run.smithi031.stdout:Signature   : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM  : grub2-2.06-68.el9.src.rpm
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date  : Wed 02 Aug 2023 03:15:48 PM UTC
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host  : x86-05.stream.rdu2.redhat.com
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:Packager    : builder@centos.org
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor      : CentOS
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:URL         : http://www.gnu.org/software/grub/
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:Summary     : Support tools for GRUB.
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:Description :
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture.  It supports a rich
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices.
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:38:20.342 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms.
2024-05-08T06:38:20.343 INFO:teuthology.task.kernel:Updating Grub Version: grub2
2024-05-08T06:38:20.343 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 5.14.0-444.el9.x86_64
2024-05-08T06:38:20.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg
2024-05-08T06:38:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ...
2024-05-08T06:38:22.439 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ...
2024-05-08T06:38:22.466 INFO:teuthology.orchestra.run.smithi031.stderr:done
2024-05-08T06:38:22.469 DEBUG:teuthology.orchestra.run.smithi031:> mktemp
2024-05-08T06:38:22.485 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.xc26XlV5b6
2024-05-08T06:38:22.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xc26XlV5b6
2024-05-08T06:38:22.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.xc26XlV5b6
2024-05-08T06:38:22.691 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.xc26XlV5b6 is 6KB
2024-05-08T06:38:22.740 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.xc26XlV5b6
2024-05-08T06:38:22.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true
2024-05-08T06:38:22.817 INFO:teuthology.orchestra.run.smithi031.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf
2024-05-08T06:38:22.818 INFO:teuthology.orchestra.run.smithi031.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf
2024-05-08T06:38:22.818 INFO:teuthology.orchestra.run.smithi031.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf
2024-05-08T06:38:22.818 INFO:teuthology.orchestra.run.smithi031.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf
2024-05-08T06:38:22.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64
2024-05-08T06:38:22.952 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now
2024-05-08T06:38:27.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83
2024-05-08T06:38:36.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com'
2024-05-08T06:38:36.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:38:39.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83
2024-05-08T06:38:51.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com'
2024-05-08T06:38:51.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:38:52.956 INFO:teuthology.task.kernel:Checking client host.a for new kernel version...
2024-05-08T06:38:52.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com'
2024-05-08T06:38:52.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:39:11.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31
2024-05-08T06:39:20.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com'
2024-05-08T06:39:20.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:39:23.318 DEBUG:teuthology.orchestra.run.smithi083:> true
2024-05-08T06:39:23.657 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com'
2024-05-08T06:39:23.657 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-444.el9.x86_64"...
2024-05-08T06:39:23.657 DEBUG:teuthology.orchestra.run.smithi083:> uname -r
2024-05-08T06:39:23.672 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-444.el9.x86_64
2024-05-08T06:39:23.672 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64
2024-05-08T06:39:23.672 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install
2024-05-08T06:39:23.672 DEBUG:teuthology.task.kernel:Distro of this test job: centos
2024-05-08T06:39:24.673 INFO:teuthology.task.kernel:Enabling kdb on host.b...
2024-05-08T06:39:24.673 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc
2024-05-08T06:39:24.744 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1
2024-05-08T06:39:24.787 DEBUG:teuthology.parallel:result is None
2024-05-08T06:39:38.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31
2024-05-08T06:39:50.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com'
2024-05-08T06:39:50.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:39:51.151 DEBUG:teuthology.orchestra.run.smithi031:> true
2024-05-08T06:39:51.588 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com'
2024-05-08T06:39:51.589 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-444.el9.x86_64"...
2024-05-08T06:39:51.589 DEBUG:teuthology.orchestra.run.smithi031:> uname -r
2024-05-08T06:39:51.605 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-444.el9.x86_64
2024-05-08T06:39:51.605 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64
2024-05-08T06:39:51.605 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install
2024-05-08T06:39:51.605 DEBUG:teuthology.task.kernel:Distro of this test job: centos
2024-05-08T06:39:52.606 INFO:teuthology.task.kernel:Enabling kdb on host.a...
2024-05-08T06:39:52.607 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc
2024-05-08T06:39:52.700 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1
2024-05-08T06:39:52.736 DEBUG:teuthology.parallel:result is None
2024-05-08T06:39:52.736 INFO:teuthology.run_tasks:Running task internal.base...
2024-05-08T06:39:52.743 INFO:teuthology.task.internal:Creating test directory...
2024-05-08T06:39:52.744 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest
2024-05-08T06:39:52.746 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest
2024-05-08T06:39:52.765 INFO:teuthology.run_tasks:Running task internal.archive_upload...
2024-05-08T06:39:52.772 INFO:teuthology.run_tasks:Running task internal.archive...
2024-05-08T06:39:52.779 INFO:teuthology.task.internal:Creating archive directory...
2024-05-08T06:39:52.779 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive
2024-05-08T06:39:52.802 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive
2024-05-08T06:39:52.849 INFO:teuthology.run_tasks:Running task internal.coredump...
2024-05-08T06:39:52.857 INFO:teuthology.task.internal:Enabling coredump saving...
2024-05-08T06:39:52.857 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf
2024-05-08T06:39:52.864 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf
2024-05-08T06:39:52.900 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core
2024-05-08T06:39:52.910 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core
2024-05-08T06:39:52.922 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core
2024-05-08T06:39:52.934 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core
2024-05-08T06:39:52.935 INFO:teuthology.run_tasks:Running task internal.sudo...
2024-05-08T06:39:52.943 INFO:teuthology.task.internal:Configuring sudo...
2024-05-08T06:39:52.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers
2024-05-08T06:39:52.955 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers
2024-05-08T06:39:53.002 INFO:teuthology.run_tasks:Running task internal.syslog...
2024-05-08T06:39:53.011 INFO:teuthology.task.internal.syslog:Starting syslog monitoring...
2024-05-08T06:39:53.012 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog
2024-05-08T06:39:53.020 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog
2024-05-08T06:39:53.060 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log
2024-05-08T06:39:53.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log
2024-05-08T06:39:53.164 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log
2024-05-08T06:39:53.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log
2024-05-08T06:39:53.286 DEBUG:teuthology.orchestra.run.smithi031:> set -ex
2024-05-08T06:39:53.287 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf
2024-05-08T06:39:53.352 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log
2024-05-08T06:39:53.388 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log
2024-05-08T06:39:53.441 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log
2024-05-08T06:39:53.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log
2024-05-08T06:39:53.565 DEBUG:teuthology.orchestra.run.smithi083:> set -ex
2024-05-08T06:39:53.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf
2024-05-08T06:39:53.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart
2024-05-08T06:39:53.633 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart
2024-05-08T06:39:53.679 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service
2024-05-08T06:39:53.721 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service
2024-05-08T06:39:54.152 INFO:teuthology.run_tasks:Running task internal.timer...
2024-05-08T06:39:54.159 INFO:teuthology.task.internal:Starting timer...
2024-05-08T06:39:54.159 INFO:teuthology.run_tasks:Running task pcp...
2024-05-08T06:39:54.169 INFO:teuthology.run_tasks:Running task selinux...
2024-05-08T06:39:54.178 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}
2024-05-08T06:39:54.179 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate
2024-05-08T06:39:54.247 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs:
2024-05-08T06:39:54.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate
2024-05-08T06:39:54.300 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs:
2024-05-08T06:39:54.302 DEBUG:teuthology.task.selinux:Getting current SELinux state
2024-05-08T06:39:54.303 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce
2024-05-08T06:39:54.343 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive
2024-05-08T06:39:54.343 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce
2024-05-08T06:39:54.367 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive
2024-05-08T06:39:54.367 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi083.front.sepia.ceph.com': 'permissive'}
2024-05-08T06:39:54.368 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0
2024-05-08T06:39:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:39:54.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0
2024-05-08T06:39:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:39:54.435 INFO:teuthology.task.selinux:Putting SELinux into permissive mode
2024-05-08T06:39:54.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive
2024-05-08T06:39:54.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive
2024-05-08T06:39:54.511 INFO:teuthology.run_tasks:Running task ansible.cephlab...
2024-05-08T06:39:54.520 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main
2024-05-08T06:39:54.628 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}]
2024-05-08T06:39:54.629 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi083.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml
2024-05-08T06:43:57.449 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi083.front.sepia.ceph.com')]
2024-05-08T06:43:57.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com'
2024-05-08T06:43:57.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:43:57.529 DEBUG:teuthology.orchestra.run.smithi031:> true
2024-05-08T06:43:57.568 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com'
2024-05-08T06:43:57.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com'
2024-05-08T06:43:57.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-05-08T06:43:57.642 DEBUG:teuthology.orchestra.run.smithi083:> true
2024-05-08T06:43:57.717 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com'
2024-05-08T06:43:57.717 INFO:teuthology.run_tasks:Running task clock...
2024-05-08T06:43:57.728 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew...
2024-05-08T06:43:57.729 INFO:teuthology.orchestra.run:Running command with timeout 360
2024-05-08T06:43:57.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true
2024-05-08T06:43:57.732 INFO:teuthology.orchestra.run:Running command with timeout 360
2024-05-08T06:43:57.732 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true
2024-05-08T06:43:57.762 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded.
2024-05-08T06:43:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded.
2024-05-08T06:43:57.786 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded.
2024-05-08T06:43:57.799 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded.
2024-05-08T06:43:57.804 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found
2024-05-08T06:43:57.816 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon
2024-05-08T06:43:57.823 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found
2024-05-08T06:43:57.831 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found.
2024-05-08T06:43:57.834 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon
2024-05-08T06:43:57.845 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found.
2024-05-08T06:43:57.848 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found.
2024-05-08T06:43:57.861 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found.
2024-05-08T06:43:57.903 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found
2024-05-08T06:43:57.908 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address         Stratum Poll Reach LastRx Last sample
2024-05-08T06:43:57.908 INFO:teuthology.orchestra.run.smithi031.stdout:===============================================================================
2024-05-08T06:43:57.908 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com     0   6     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:43:57.908 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com     0   6     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:43:57.908 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com     0   6     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:43:57.909 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com     0   6     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:43:57.914 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found
2024-05-08T06:43:57.919 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address         Stratum Poll Reach LastRx Last sample
2024-05-08T06:43:57.919 INFO:teuthology.orchestra.run.smithi083.stdout:===============================================================================
2024-05-08T06:43:57.919 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com     0   6     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:43:57.919 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com     0   6     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:43:57.920 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com     0   6     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:43:57.920 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com     0   6     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:43:57.920 INFO:teuthology.run_tasks:Running task pexec...
2024-05-08T06:43:57.934 INFO:teuthology.task.pexec:Executing custom commands...
2024-05-08T06:43:57.934 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi031.front.sepia.ceph.com
2024-05-08T06:43:57.934 DEBUG:teuthology.orchestra.run.smithi031:> TESTDIR=/home/ubuntu/cephtest bash -s
2024-05-08T06:43:57.935 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi083.front.sepia.ceph.com
2024-05-08T06:43:57.935 DEBUG:teuthology.orchestra.run.smithi083:> TESTDIR=/home/ubuntu/cephtest bash -s
2024-05-08T06:43:58.543 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:48 ago on Wed 08 May 2024 06:43:10 AM UTC.
2024-05-08T06:43:58.557 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:43 ago on Wed 08 May 2024 06:43:15 AM UTC.
2024-05-08T06:43:58.661 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved.
2024-05-08T06:43:58.677 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================
2024-05-08T06:43:58.677 INFO:teuthology.orchestra.run.smithi083.stdout: Package       Architecture    Version                 Repository          Size
2024-05-08T06:43:58.677 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================
2024-05-08T06:43:58.677 INFO:teuthology.orchestra.run.smithi083.stdout:Installing:
2024-05-08T06:43:58.677 INFO:teuthology.orchestra.run.smithi083.stdout: runc          x86_64          4:1.1.12-2.el9          appstream          3.1 M
2024-05-08T06:43:58.677 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:43:58.677 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary
2024-05-08T06:43:58.677 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================
2024-05-08T06:43:58.678 INFO:teuthology.orchestra.run.smithi083.stdout:Install  1 Package
2024-05-08T06:43:58.678 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:43:58.682 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.1 M
2024-05-08T06:43:58.682 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 9.8 M
2024-05-08T06:43:58.682 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages:
2024-05-08T06:43:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved.
2024-05-08T06:43:58.694 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================
2024-05-08T06:43:58.694 INFO:teuthology.orchestra.run.smithi031.stdout: Package       Architecture    Version                 Repository          Size
2024-05-08T06:43:58.694 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================
2024-05-08T06:43:58.694 INFO:teuthology.orchestra.run.smithi031.stdout:Installing:
2024-05-08T06:43:58.694 INFO:teuthology.orchestra.run.smithi031.stdout: runc          x86_64          4:1.1.12-2.el9          appstream          3.1 M
2024-05-08T06:43:58.694 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:43:58.694 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary
2024-05-08T06:43:58.694 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================
2024-05-08T06:43:58.694 INFO:teuthology.orchestra.run.smithi031.stdout:Install  1 Package
2024-05-08T06:43:58.695 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:43:58.704 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.1 M
2024-05-08T06:43:58.704 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 9.8 M
2024-05-08T06:43:58.704 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages:
2024-05-08T06:43:59.038 INFO:teuthology.orchestra.run.smithi083.stdout:runc-1.1.12-2.el9.x86_64.rpm                     11 MB/s | 3.1 MB     00:00
2024-05-08T06:43:59.038 INFO:teuthology.orchestra.run.smithi083.stdout:--------------------------------------------------------------------------------
2024-05-08T06:43:59.038 INFO:teuthology.orchestra.run.smithi083.stdout:Total                                           8.8 MB/s | 3.1 MB     00:00
2024-05-08T06:43:59.074 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check
2024-05-08T06:43:59.080 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded.
2024-05-08T06:43:59.080 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test
2024-05-08T06:43:59.174 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded.
2024-05-08T06:43:59.174 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction
2024-05-08T06:43:59.388 INFO:teuthology.orchestra.run.smithi083.stdout:  Preparing        :                                                        1/1
2024-05-08T06:43:59.482 INFO:teuthology.orchestra.run.smithi083.stdout:  Installing       : runc-4:1.1.12-2.el9.x86_64                             1/1
2024-05-08T06:44:00.094 INFO:teuthology.orchestra.run.smithi083.stdout:  Running scriptlet: runc-4:1.1.12-2.el9.x86_64                             1/1
2024-05-08T06:44:00.114 INFO:teuthology.orchestra.run.smithi031.stdout:runc-1.1.12-2.el9.x86_64.rpm                    2.8 MB/s | 3.1 MB     00:01
2024-05-08T06:44:00.114 INFO:teuthology.orchestra.run.smithi031.stdout:--------------------------------------------------------------------------------
2024-05-08T06:44:00.114 INFO:teuthology.orchestra.run.smithi031.stdout:Total                                           2.2 MB/s | 3.1 MB     00:01
2024-05-08T06:44:00.154 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check
2024-05-08T06:44:00.161 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded.
2024-05-08T06:44:00.161 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test
2024-05-08T06:44:00.272 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded.
2024-05-08T06:44:00.272 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction
2024-05-08T06:44:00.366 INFO:teuthology.orchestra.run.smithi083.stdout:  Verifying        : runc-4:1.1.12-2.el9.x86_64                             1/1
2024-05-08T06:44:00.366 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:44:00.366 INFO:teuthology.orchestra.run.smithi083.stdout:Installed:
2024-05-08T06:44:00.367 INFO:teuthology.orchestra.run.smithi083.stdout:  runc-4:1.1.12-2.el9.x86_64
2024-05-08T06:44:00.367 INFO:teuthology.orchestra.run.smithi083.stdout:
2024-05-08T06:44:00.367 INFO:teuthology.orchestra.run.smithi083.stdout:Complete!
2024-05-08T06:44:00.505 INFO:teuthology.orchestra.run.smithi031.stdout:  Preparing        :                                                        1/1
2024-05-08T06:44:00.594 INFO:teuthology.orchestra.run.smithi031.stdout:  Installing       : runc-4:1.1.12-2.el9.x86_64                             1/1
2024-05-08T06:44:00.666 DEBUG:teuthology.parallel:result is None
2024-05-08T06:44:01.111 INFO:teuthology.orchestra.run.smithi031.stdout:  Running scriptlet: runc-4:1.1.12-2.el9.x86_64                             1/1
2024-05-08T06:44:01.362 INFO:teuthology.orchestra.run.smithi031.stdout:  Verifying        : runc-4:1.1.12-2.el9.x86_64                             1/1
2024-05-08T06:44:01.363 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:44:01.363 INFO:teuthology.orchestra.run.smithi031.stdout:Installed:
2024-05-08T06:44:01.363 INFO:teuthology.orchestra.run.smithi031.stdout:  runc-4:1.1.12-2.el9.x86_64
2024-05-08T06:44:01.363 INFO:teuthology.orchestra.run.smithi031.stdout:
2024-05-08T06:44:01.363 INFO:teuthology.orchestra.run.smithi031.stdout:Complete!
2024-05-08T06:44:01.652 DEBUG:teuthology.parallel:result is None
2024-05-08T06:44:01.652 INFO:teuthology.run_tasks:Running task nvme_loop...
2024-05-08T06:44:01.663 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices...
2024-05-08T06:44:01.663 DEBUG:teuthology.orchestra.run.smithi031:> set -ex
2024-05-08T06:44:01.663 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout
2024-05-08T06:44:01.680 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4']
2024-05-08T06:44:01.680 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1
2024-05-08T06:44:01.735 INFO:teuthology.orchestra.run.smithi031.stdout:  File: /dev/vg_nvme/lv_1 -> ../dm-0
2024-05-08T06:44:01.736 INFO:teuthology.orchestra.run.smithi031.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T06:44:01.736 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d	Inode: 667         Links: 1
2024-05-08T06:44:01.736 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T06:44:01.736 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0
2024-05-08T06:44:01.736 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-05-08 06:43:56.413101251 +0000
2024-05-08T06:44:01.736 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-05-08 06:43:38.593383248 +0000
2024-05-08T06:44:01.736 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-05-08 06:43:38.593383248 +0000
2024-05-08T06:44:01.736 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-05-08 06:43:38.593383248 +0000
2024-05-08T06:44:01.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1
2024-05-08T06:44:01.799 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in
2024-05-08T06:44:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out
2024-05-08T06:44:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000203647 s, 2.5 MB/s
2024-05-08T06:44:01.801 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1
2024-05-08T06:44:01.858 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2
2024-05-08T06:44:01.915 INFO:teuthology.orchestra.run.smithi031.stdout:  File: /dev/vg_nvme/lv_2 -> ../dm-1
2024-05-08T06:44:01.915 INFO:teuthology.orchestra.run.smithi031.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T06:44:01.915 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d	Inode: 684         Links: 1
2024-05-08T06:44:01.915 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T06:44:01.915 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0
2024-05-08T06:44:01.915 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-05-08 06:43:56.413101251 +0000
2024-05-08T06:44:01.915 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-05-08 06:43:38.953377551 +0000
2024-05-08T06:44:01.915 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-05-08 06:43:38.953377551 +0000
2024-05-08T06:44:01.916 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-05-08 06:43:38.953377551 +0000
2024-05-08T06:44:01.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1
2024-05-08T06:44:01.981 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in
2024-05-08T06:44:01.981 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out
2024-05-08T06:44:01.981 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000185717 s, 2.8 MB/s
2024-05-08T06:44:01.982 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2
2024-05-08T06:44:02.039 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3
2024-05-08T06:44:02.097 INFO:teuthology.orchestra.run.smithi031.stdout:  File: /dev/vg_nvme/lv_3 -> ../dm-2
2024-05-08T06:44:02.097 INFO:teuthology.orchestra.run.smithi031.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T06:44:02.098 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d	Inode: 703         Links: 1
2024-05-08T06:44:02.098 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T06:44:02.098 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0
2024-05-08T06:44:02.098 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-05-08 06:43:56.413101251 +0000
2024-05-08T06:44:02.098 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-05-08 06:43:39.323371696 +0000
2024-05-08T06:44:02.098 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-05-08 06:43:39.323371696 +0000
2024-05-08T06:44:02.098 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-05-08 06:43:39.323371696 +0000
2024-05-08T06:44:02.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1
2024-05-08T06:44:02.162 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in
2024-05-08T06:44:02.162 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out
2024-05-08T06:44:02.162 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000185289 s, 2.8 MB/s
2024-05-08T06:44:02.164 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3
2024-05-08T06:44:02.221 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4
2024-05-08T06:44:02.277 INFO:teuthology.orchestra.run.smithi031.stdout:  File: /dev/vg_nvme/lv_4 -> ../dm-3
2024-05-08T06:44:02.278 INFO:teuthology.orchestra.run.smithi031.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T06:44:02.278 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d	Inode: 725         Links: 1
2024-05-08T06:44:02.278 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T06:44:02.278 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0
2024-05-08T06:44:02.278 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-05-08 06:43:56.413101251 +0000
2024-05-08T06:44:02.278 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-05-08 06:43:39.665366283 +0000
2024-05-08T06:44:02.278 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-05-08 06:43:39.665366283 +0000
2024-05-08T06:44:02.278 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-05-08 06:43:39.665366283 +0000
2024-05-08T06:44:02.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1
2024-05-08T06:44:02.342 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in
2024-05-08T06:44:02.342 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out
2024-05-08T06:44:02.342 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000189522 s, 2.7 MB/s
2024-05-08T06:44:02.344 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4
2024-05-08T06:44:02.400 DEBUG:teuthology.orchestra.run.smithi031:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype
2024-05-08T06:44:02.557 INFO:teuthology.orchestra.run.smithi031.stdout:loop
2024-05-08T06:44:02.558 INFO:tasks.nvme_loop:Connecting nvme_loop smithi031:/dev/vg_nvme/lv_1...
2024-05-08T06:44:02.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn
2024-05-08T06:44:02.593 INFO:teuthology.orchestra.run.smithi031.stdout:1
2024-05-08T06:44:02.624 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/vg_nvme/lv_11
2024-05-08T06:44:02.657 INFO:tasks.nvme_loop:Connecting nvme_loop smithi031:/dev/vg_nvme/lv_2...
2024-05-08T06:44:02.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn
2024-05-08T06:44:02.731 INFO:teuthology.orchestra.run.smithi031.stdout:1
2024-05-08T06:44:02.762 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/vg_nvme/lv_21
2024-05-08T06:44:02.786 INFO:tasks.nvme_loop:Connecting nvme_loop smithi031:/dev/vg_nvme/lv_3...
2024-05-08T06:44:02.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn
2024-05-08T06:44:02.860 INFO:teuthology.orchestra.run.smithi031.stdout:1
2024-05-08T06:44:02.890 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/vg_nvme/lv_31
2024-05-08T06:44:02.914 INFO:tasks.nvme_loop:Connecting nvme_loop smithi031:/dev/vg_nvme/lv_4...
2024-05-08T06:44:02.915 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn
2024-05-08T06:44:02.990 INFO:teuthology.orchestra.run.smithi031.stdout:1
2024-05-08T06:44:03.020 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/vg_nvme/lv_41
2024-05-08T06:44:03.052 DEBUG:teuthology.orchestra.run.smithi031:> set -ex
2024-05-08T06:44:03.052 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout
2024-05-08T06:44:03.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo nvme list
2024-05-08T06:44:03.173 INFO:teuthology.orchestra.run.smithi031.stdout:Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
2024-05-08T06:44:03.173 INFO:teuthology.orchestra.run.smithi031.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
2024-05-08T06:44:03.173 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/nvme0n1     CVFT53310008400BGN   INTEL SSDPEDMD400G4                      1         400.09  GB / 400.09  GB    512   B +  0 B   8DV101H0
2024-05-08T06:44:03.173 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/nvme1n1     5ca876e7120ed180ab17 Linux                                    1          96.00  GB /  96.00  GB    512   B +  0 B   5.14.0-4
2024-05-08T06:44:03.173 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/nvme2n1     2f069227fa4af261f06a Linux                                    1          96.00  GB /  96.00  GB    512   B +  0 B   5.14.0-4
2024-05-08T06:44:03.173 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/nvme3n1     dc1a70ad4d1e458776fb Linux                                    1          96.00  GB /  96.00  GB    512   B +  0 B   5.14.0-4
2024-05-08T06:44:03.173 INFO:teuthology.orchestra.run.smithi031.stdout:/dev/nvme4n1     9a90e19230db4cd22b87 Linux                                    1          96.00  GB /  96.00  GB    512   B +  0 B   5.14.0-4
2024-05-08T06:44:03.174 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1']
2024-05-08T06:44:03.175 DEBUG:teuthology.orchestra.run.smithi031:> set -ex
2024-05-08T06:44:03.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/scratch_devs
2024-05-08T06:44:03.241 DEBUG:teuthology.orchestra.run.smithi083:> set -ex
2024-05-08T06:44:03.241 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout
2024-05-08T06:44:03.261 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4']
2024-05-08T06:44:03.261 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1
2024-05-08T06:44:03.321 INFO:teuthology.orchestra.run.smithi083.stdout:  File: /dev/vg_nvme/lv_1 -> ../dm-0
2024-05-08T06:44:03.321 INFO:teuthology.orchestra.run.smithi083.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T06:44:03.321 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d	Inode: 665         Links: 1
2024-05-08T06:44:03.321 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T06:44:03.321 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0
2024-05-08T06:44:03.321 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-05-08 06:43:56.417095932 +0000
2024-05-08T06:44:03.322 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-05-08 06:43:33.143683803 +0000
2024-05-08T06:44:03.322 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-05-08 06:43:33.143683803 +0000
2024-05-08T06:44:03.322 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-05-08 06:43:33.143683803 +0000
2024-05-08T06:44:03.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1
2024-05-08T06:44:03.391 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in
2024-05-08T06:44:03.392 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out
2024-05-08T06:44:03.392 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000290018 s, 1.8 MB/s
2024-05-08T06:44:03.393 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1
2024-05-08T06:44:03.452 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2
2024-05-08T06:44:03.510 INFO:teuthology.orchestra.run.smithi083.stdout:  File: /dev/vg_nvme/lv_2 -> ../dm-1
2024-05-08T06:44:03.510 INFO:teuthology.orchestra.run.smithi083.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T06:44:03.510 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d	Inode: 685         Links: 1
2024-05-08T06:44:03.510 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T06:44:03.510 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0
2024-05-08T06:44:03.510 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-05-08 06:43:56.417095932 +0000
2024-05-08T06:44:03.510 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-05-08 06:43:33.511674443 +0000
2024-05-08T06:44:03.511 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-05-08 06:43:33.511674443 +0000
2024-05-08T06:44:03.511 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-05-08 06:43:33.511674443 +0000
2024-05-08T06:44:03.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1
2024-05-08T06:44:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in
2024-05-08T06:44:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out
2024-05-08T06:44:03.576 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000184673 s, 2.8 MB/s
2024-05-08T06:44:03.578 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2
2024-05-08T06:44:03.635 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3
2024-05-08T06:44:03.692 INFO:teuthology.orchestra.run.smithi083.stdout:  File: /dev/vg_nvme/lv_3 -> ../dm-2
2024-05-08T06:44:03.692 INFO:teuthology.orchestra.run.smithi083.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T06:44:03.692 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d	Inode: 704         Links: 1
2024-05-08T06:44:03.693 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T06:44:03.693 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0
2024-05-08T06:44:03.693 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-05-08 06:43:56.417095932 +0000
2024-05-08T06:44:03.693 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-05-08 06:43:33.896664651 +0000
2024-05-08T06:44:03.693 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-05-08 06:43:33.896664651 +0000
2024-05-08T06:44:03.693 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-05-08 06:43:33.896664651 +0000
2024-05-08T06:44:03.693 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1
2024-05-08T06:44:03.758 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in
2024-05-08T06:44:03.758 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out
2024-05-08T06:44:03.758 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000193308 s, 2.6 MB/s
2024-05-08T06:44:03.759 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3
2024-05-08T06:44:03.817 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4
2024-05-08T06:44:03.873 INFO:teuthology.orchestra.run.smithi083.stdout:  File: /dev/vg_nvme/lv_4 -> ../dm-3
2024-05-08T06:44:03.873 INFO:teuthology.orchestra.run.smithi083.stdout:  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
2024-05-08T06:44:03.873 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d	Inode: 725         Links: 1
2024-05-08T06:44:03.873 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2024-05-08T06:44:03.873 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0
2024-05-08T06:44:03.873 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-05-08 06:43:56.417095932 +0000
2024-05-08T06:44:03.873 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-05-08 06:43:34.229656181 +0000
2024-05-08T06:44:03.873 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-05-08 06:43:34.229656181 +0000
2024-05-08T06:44:03.873 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-05-08 06:43:34.229656181 +0000
2024-05-08T06:44:03.873 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1
2024-05-08T06:44:03.937 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in
2024-05-08T06:44:03.938 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out
2024-05-08T06:44:03.938 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000200442 s, 2.6 MB/s
2024-05-08T06:44:03.939 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4
2024-05-08T06:44:03.995 DEBUG:teuthology.orchestra.run.smithi083:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype
2024-05-08T06:44:04.131 INFO:teuthology.orchestra.run.smithi083.stdout:loop
2024-05-08T06:44:04.132 INFO:tasks.nvme_loop:Connecting nvme_loop smithi083:/dev/vg_nvme/lv_1...
2024-05-08T06:44:04.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn
2024-05-08T06:44:04.165 INFO:teuthology.orchestra.run.smithi083.stdout:1
2024-05-08T06:44:04.194 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/vg_nvme/lv_11
2024-05-08T06:44:04.217 INFO:tasks.nvme_loop:Connecting nvme_loop smithi083:/dev/vg_nvme/lv_2...
2024-05-08T06:44:04.218 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn
2024-05-08T06:44:04.291 INFO:teuthology.orchestra.run.smithi083.stdout:1
2024-05-08T06:44:04.320 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/vg_nvme/lv_21
2024-05-08T06:44:04.344 INFO:tasks.nvme_loop:Connecting nvme_loop smithi083:/dev/vg_nvme/lv_3...
2024-05-08T06:44:04.344 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn
2024-05-08T06:44:04.423 INFO:teuthology.orchestra.run.smithi083.stdout:1
2024-05-08T06:44:04.452 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/vg_nvme/lv_31
2024-05-08T06:44:04.475 INFO:tasks.nvme_loop:Connecting nvme_loop smithi083:/dev/vg_nvme/lv_4...
2024-05-08T06:44:04.475 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn
2024-05-08T06:44:04.553 INFO:teuthology.orchestra.run.smithi083.stdout:1
2024-05-08T06:44:04.582 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/vg_nvme/lv_41
2024-05-08T06:44:04.605 DEBUG:teuthology.orchestra.run.smithi083:> set -ex
2024-05-08T06:44:04.605 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout
2024-05-08T06:44:04.666 DEBUG:teuthology.orchestra.run.smithi083:> sudo nvme list
2024-05-08T06:44:04.729 INFO:teuthology.orchestra.run.smithi083.stdout:Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
2024-05-08T06:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
2024-05-08T06:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/nvme0n1     PHFT62040148400BGN   INTEL SSDPEDMD400G4                      1         400.09  GB / 400.09  GB    512   B +  0 B   8DV101H0
2024-05-08T06:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/nvme1n1     a7443ddbbc897b25ae9c Linux                                    1          96.00  GB /  96.00  GB    512   B +  0 B   5.14.0-4
2024-05-08T06:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/nvme2n1     5eb96bd4159256bd64c5 Linux                                    1          96.00  GB /  96.00  GB    512   B +  0 B   5.14.0-4
2024-05-08T06:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/nvme3n1     86cb56cdc8b2641bd995 Linux                                    1          96.00  GB /  96.00  GB    512   B +  0 B   5.14.0-4
2024-05-08T06:44:04.730 INFO:teuthology.orchestra.run.smithi083.stdout:/dev/nvme4n1     203a3aa183b8941b5a8d Linux                                    1          96.00  GB /  96.00  GB    512   B +  0 B   5.14.0-4
2024-05-08T06:44:04.731 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1']
2024-05-08T06:44:04.731 DEBUG:teuthology.orchestra.run.smithi083:> set -ex
2024-05-08T06:44:04.731 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/scratch_devs
2024-05-08T06:44:04.795 INFO:teuthology.run_tasks:Running task cephadm...
2024-05-08T06:44:04.897 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '7c8f650b36e258f639fa4a83becade57cbfd2009'}
2024-05-08T06:44:04.897 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009
2024-05-08T06:44:04.898 INFO:tasks.cephadm:Cluster fsid is 5cc2602c-0d06-11ef-bc97-c7b262605968
2024-05-08T06:44:04.898 INFO:tasks.cephadm:Choosing monitor IPs and ports...
2024-05-08T06:44:04.898 INFO:tasks.cephadm:No mon roles; fabricating mons
2024-05-08T06:44:04.898 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi031': '172.21.15.31', 'mon.smithi083': '172.21.15.83'}
2024-05-08T06:44:04.898 INFO:tasks.cephadm:Normalizing hostnames...
2024-05-08T06:44:04.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s)
2024-05-08T06:44:04.930 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname $(hostname -s)
2024-05-08T06:44:04.955 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra
2024-05-08T06:44:04.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7c8f650b36e258f639fa4a83becade57cbfd2009
2024-05-08T06:44:05.180 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '7c8f650b36e258f639fa4a83becade57cbfd2009', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79172/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3593-g7c8f650b', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3593.g7c8f650b'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-05-07-0802/7c8f650b36e258f639fa4a83becade57cbfd2009/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-07 17:59:41.144528', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-05-07-0802', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-05-07-0802/7c8f650b36e258f639fa4a83becade57cbfd2009/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}]
2024-05-08T06:44:05.323 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-adk-testing-2024-05-07-0802, sha1 7c8f650b36e258f639fa4a83becade57cbfd2009 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=7c8f650b36e258f639fa4a83becade57cbfd2009
2024-05-08T06:44:05.325 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-05-07-0802/7c8f650b36e258f639fa4a83becade57cbfd2009/centos/9/x86_64/flavors/default/cephadm
2024-05-08T06:44:05.325 INFO:tasks.cephadm:Downloading cephadm from url: https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-05-07-0802/7c8f650b36e258f639fa4a83becade57cbfd2009/centos/9/x86_64/flavors/default/cephadm
2024-05-08T06:44:05.325 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-05-07-0802/7c8f650b36e258f639fa4a83becade57cbfd2009/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm
2024-05-08T06:44:05.903 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r--. 1 ubuntu ubuntu 783059 May  8 06:44 /home/ubuntu/cephtest/cephadm
2024-05-08T06:44:05.903 DEBUG:teuthology.orchestra.run.smithi083:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-05-07-0802/7c8f650b36e258f639fa4a83becade57cbfd2009/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm
2024-05-08T06:44:06.332 INFO:teuthology.orchestra.run.smithi083.stdout:-rw-r--r--. 1 ubuntu ubuntu 783059 May  8 06:44 /home/ubuntu/cephtest/cephadm
2024-05-08T06:44:06.332 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm
2024-05-08T06:44:06.353 DEBUG:teuthology.orchestra.run.smithi083:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm
2024-05-08T06:44:06.381 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009 on all hosts...
2024-05-08T06:44:06.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009 pull
2024-05-08T06:44:06.395 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009 pull
2024-05-08T06:44:07.904 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009...
2024-05-08T06:44:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009...
2024-05-08T06:44:08.104 INFO:teuthology.orchestra.run.smithi031.stderr:Non-zero exit code 125 from /bin/podman pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009
2024-05-08T06:44:08.104 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009...
2024-05-08T06:44:08.104 INFO:teuthology.orchestra.run.smithi031.stderr:/bin/podman: stderr Error: initializing source docker://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009: reading manifest 7c8f650b36e258f639fa4a83becade57cbfd2009 in quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph: manifest unknown
2024-05-08T06:44:08.104 INFO:teuthology.orchestra.run.smithi031.stderr:ERROR: Failed command: /bin/podman pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009
2024-05-08T06:44:08.128 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:08.128 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa/tasks/cephadm.py", line 585, in pull_image
    run.wait(ctx.cluster.run(args=cmd, wait=False))
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009 pull'
2024-05-08T06:44:08.130 INFO:tasks.cephadm:Archiving crash dumps...
2024-05-08T06:44:08.132 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/5cc2602c-0d06-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333/remote/smithi031/crash
2024-05-08T06:44:08.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/5cc2602c-0d06-11ef-bc97-c7b262605968/crash -- .
2024-05-08T06:44:08.155 INFO:teuthology.orchestra.run.smithi083.stderr:Non-zero exit code 125 from /bin/podman pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009
2024-05-08T06:44:08.155 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009...
2024-05-08T06:44:08.155 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: stderr Error: initializing source docker://quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009: reading manifest 7c8f650b36e258f639fa4a83becade57cbfd2009 in quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph: manifest unknown
2024-05-08T06:44:08.156 INFO:teuthology.orchestra.run.smithi083.stderr:ERROR: Failed command: /bin/podman pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009
2024-05-08T06:44:08.168 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/5cc2602c-0d06-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory
2024-05-08T06:44:08.168 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now
2024-05-08T06:44:08.170 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/5cc2602c-0d06-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333/remote/smithi083/crash
2024-05-08T06:44:08.171 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/5cc2602c-0d06-11ef-bc97-c7b262605968/crash -- .
2024-05-08T06:44:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:tar: /var/lib/ceph/5cc2602c-0d06-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory
2024-05-08T06:44:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:tar: Error is not recoverable: exiting now
2024-05-08T06:44:08.209 INFO:tasks.cephadm:Checking cluster log for badness...
2024-05-08T06:44:08.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5cc2602c-0d06-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1
2024-05-08T06:44:08.236 INFO:teuthology.orchestra.run.smithi031.stderr:grep: /var/log/ceph/5cc2602c-0d06-11ef-bc97-c7b262605968/ceph.log: No such file or directory
2024-05-08T06:44:08.238 INFO:tasks.cephadm:Compressing logs...
2024-05-08T06:44:08.238 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2024-05-08T06:44:08.281 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2024-05-08T06:44:08.304 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory
2024-05-08T06:44:08.305 INFO:teuthology.orchestra.run.smithi031.stderr: -- /var/log/ceph/cephadm.log
2024-05-08T06:44:08.305 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cephadm.log:	 73.5% -- replaced with /var/log/ceph/cephadm.log.gz
2024-05-08T06:44:08.306 INFO:teuthology.orchestra.run.smithi083.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory
2024-05-08T06:44:08.306 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/cephadm.log
2024-05-08T06:44:08.306 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/cephadm.log:	 73.7% -- replaced with /var/log/ceph/cephadm.log.gz
2024-05-08T06:44:08.307 INFO:teuthology.orchestra.run.smithi031.stderr:
2024-05-08T06:44:08.308 INFO:teuthology.orchestra.run.smithi031.stderr:real	0m0.013s
2024-05-08T06:44:08.308 INFO:teuthology.orchestra.run.smithi031.stderr:user	0m0.009s
2024-05-08T06:44:08.308 INFO:teuthology.orchestra.run.smithi031.stderr:sys	0m0.014s
2024-05-08T06:44:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:
2024-05-08T06:44:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:real	0m0.012s
2024-05-08T06:44:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:user	0m0.013s
2024-05-08T06:44:08.309 INFO:teuthology.orchestra.run.smithi083.stderr:sys	0m0.009s
2024-05-08T06:44:08.309 INFO:tasks.cephadm:Archiving logs...
2024-05-08T06:44:08.309 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333/remote/smithi031/log
2024-05-08T06:44:08.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- .
2024-05-08T06:44:08.375 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333/remote/smithi083/log
2024-05-08T06:44:08.375 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- .
2024-05-08T06:44:08.400 INFO:tasks.cephadm:Removing cluster...
2024-05-08T06:44:08.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5cc2602c-0d06-11ef-bc97-c7b262605968 --force
2024-05-08T06:44:08.612 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 5cc2602c-0d06-11ef-bc97-c7b262605968
2024-05-08T06:44:08.733 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5cc2602c-0d06-11ef-bc97-c7b262605968 --force
2024-05-08T06:44:08.916 INFO:teuthology.orchestra.run.smithi083.stdout:Deleting cluster with fsid: 5cc2602c-0d06-11ef-bc97-c7b262605968
2024-05-08T06:44:09.039 INFO:tasks.cephadm:Removing cephadm ...
2024-05-08T06:44:09.039 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm
2024-05-08T06:44:09.055 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/cephadm
2024-05-08T06:44:09.070 INFO:tasks.cephadm:Teardown complete
2024-05-08T06:44:09.070 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa/tasks/cephadm.py", line 2244, in task
    with contextutil.nested(
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested
    raise exc[1]
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa/tasks/cephadm.py", line 1873, in initialize_config
    yield
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa/tasks/cephadm.py", line 585, in pull_image
    run.wait(ctx.cluster.run(args=cmd, wait=False))
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009 pull'
2024-05-08T06:44:09.294 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=744999b3b1aa47cba2ab17deb50e6c47
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa/tasks/cephadm.py", line 2244, in task
    with contextutil.nested(
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested
    raise exc[1]
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa/tasks/cephadm.py", line 1873, in initialize_config
    yield
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_7c8f650b36e258f639fa4a83becade57cbfd2009/qa/tasks/cephadm.py", line 585, in pull_image
    run.wait(ctx.cluster.run(args=cmd, wait=False))
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009 pull'
2024-05-08T06:44:09.298 DEBUG:teuthology.run_tasks:Unwinding manager cephadm
2024-05-08T06:44:09.307 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop
2024-05-08T06:44:09.327 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi031:/dev/vg_nvme/lv_1...
2024-05-08T06:44:09.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo nvme disconnect -n lv_1
2024-05-08T06:44:09.481 INFO:teuthology.orchestra.run.smithi031.stdout:NQN:lv_1 disconnected 1 controller(s)
2024-05-08T06:44:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:09.483 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi031:/dev/vg_nvme/lv_2...
2024-05-08T06:44:09.483 DEBUG:teuthology.orchestra.run.smithi031:> sudo nvme disconnect -n lv_2
2024-05-08T06:44:09.621 INFO:teuthology.orchestra.run.smithi031.stdout:NQN:lv_2 disconnected 1 controller(s)
2024-05-08T06:44:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:09.623 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi031:/dev/vg_nvme/lv_3...
2024-05-08T06:44:09.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo nvme disconnect -n lv_3
2024-05-08T06:44:09.768 INFO:teuthology.orchestra.run.smithi031.stdout:NQN:lv_3 disconnected 1 controller(s)
2024-05-08T06:44:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:09.769 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi031:/dev/vg_nvme/lv_4...
2024-05-08T06:44:09.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo nvme disconnect -n lv_4
2024-05-08T06:44:09.910 INFO:teuthology.orchestra.run.smithi031.stdout:NQN:lv_4 disconnected 1 controller(s)
2024-05-08T06:44:09.911 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:09.911 DEBUG:teuthology.orchestra.run.smithi031:> set -ex
2024-05-08T06:44:09.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/scratch_devs
2024-05-08T06:44:09.938 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi083:/dev/vg_nvme/lv_1...
2024-05-08T06:44:09.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo nvme disconnect -n lv_1
2024-05-08T06:44:10.075 INFO:teuthology.orchestra.run.smithi083.stdout:NQN:lv_1 disconnected 1 controller(s)
2024-05-08T06:44:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:10.076 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi083:/dev/vg_nvme/lv_2...
2024-05-08T06:44:10.077 DEBUG:teuthology.orchestra.run.smithi083:> sudo nvme disconnect -n lv_2
2024-05-08T06:44:10.214 INFO:teuthology.orchestra.run.smithi083.stdout:NQN:lv_2 disconnected 1 controller(s)
2024-05-08T06:44:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:10.215 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi083:/dev/vg_nvme/lv_3...
2024-05-08T06:44:10.216 DEBUG:teuthology.orchestra.run.smithi083:> sudo nvme disconnect -n lv_3
2024-05-08T06:44:10.369 INFO:teuthology.orchestra.run.smithi083.stdout:NQN:lv_3 disconnected 1 controller(s)
2024-05-08T06:44:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:10.370 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi083:/dev/vg_nvme/lv_4...
2024-05-08T06:44:10.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo nvme disconnect -n lv_4
2024-05-08T06:44:10.517 INFO:teuthology.orchestra.run.smithi083.stdout:NQN:lv_4 disconnected 1 controller(s)
2024-05-08T06:44:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:10.519 DEBUG:teuthology.orchestra.run.smithi083:> set -ex
2024-05-08T06:44:10.519 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/scratch_devs
2024-05-08T06:44:10.548 DEBUG:teuthology.run_tasks:Unwinding manager clock
2024-05-08T06:44:10.558 INFO:teuthology.task.clock:Checking final clock skew...
2024-05-08T06:44:10.558 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true
2024-05-08T06:44:10.561 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true
2024-05-08T06:44:10.576 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found
2024-05-08T06:44:10.582 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address         Stratum Poll Reach LastRx Last sample
2024-05-08T06:44:10.582 INFO:teuthology.orchestra.run.smithi031.stdout:===============================================================================
2024-05-08T06:44:10.582 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv01.front.sepia.ceph.com     3   6    17     6   +471us[ +498us] +/-   60ms
2024-05-08T06:44:10.582 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com     3   6    17     6  -1805us[-1805us] +/-   71ms
2024-05-08T06:44:10.582 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com     3   6    17     6  +1113us[+1113us] +/-   64ms
2024-05-08T06:44:10.582 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com     0   7     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:44:10.604 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found
2024-05-08T06:44:10.609 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address         Stratum Poll Reach LastRx Last sample
2024-05-08T06:44:10.609 INFO:teuthology.orchestra.run.smithi083.stdout:===============================================================================
2024-05-08T06:44:10.609 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv01.front.sepia.ceph.com     3   6    17     6   +504us[ +472us] +/-   60ms
2024-05-08T06:44:10.609 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv02.front.sepia.ceph.com     3   6    17     6  -1758us[-1789us] +/-   71ms
2024-05-08T06:44:10.609 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com     3   6    17     6  +1130us[+1130us] +/-   64ms
2024-05-08T06:44:10.609 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com     0   7     0     -     +0ns[   +0ns] +/-    0ns
2024-05-08T06:44:10.610 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab
2024-05-08T06:44:10.623 INFO:teuthology.task.ansible:Skipping ansible cleanup...
2024-05-08T06:44:10.623 DEBUG:teuthology.run_tasks:Unwinding manager selinux
2024-05-08T06:44:10.631 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log
2024-05-08T06:44:10.695 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log
2024-05-08T06:44:10.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0
2024-05-08T06:44:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:10.786 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0
2024-05-08T06:44:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:10.814 DEBUG:teuthology.run_tasks:Unwinding manager pcp
2024-05-08T06:44:10.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer
2024-05-08T06:44:10.832 INFO:teuthology.task.internal:Duration was 256.672769 seconds
2024-05-08T06:44:10.833 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog
2024-05-08T06:44:10.841 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring...
2024-05-08T06:44:10.842 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
2024-05-08T06:44:10.844 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
2024-05-08T06:44:10.887 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service
2024-05-08T06:44:10.894 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service
2024-05-08T06:44:11.162 INFO:teuthology.task.internal.syslog:Checking logs for errors...
2024-05-08T06:44:11.163 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com
2024-05-08T06:44:11.163 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1
2024-05-08T06:44:11.186 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com
2024-05-08T06:44:11.186 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1
2024-05-08T06:44:11.233 INFO:teuthology.task.internal.syslog:Compressing syslogs...
2024-05-08T06:44:11.233 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2024-05-08T06:44:11.236 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
2024-05-08T06:44:11.301 INFO:teuthology.task.internal.syslog:Gathering journactl -b0...
2024-05-08T06:44:11.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
2024-05-08T06:44:11.306 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
2024-05-08T06:44:11.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo
2024-05-08T06:44:11.438 INFO:teuthology.task.internal:Restoring /etc/sudoers...
2024-05-08T06:44:11.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers
2024-05-08T06:44:11.465 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers
2024-05-08T06:44:11.489 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
2024-05-08T06:44:11.502 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
2024-05-08T06:44:11.508 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
2024-05-08T06:44:11.529 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core
2024-05-08T06:44:11.552 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core
2024-05-08T06:44:11.574 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump
2024-05-08T06:44:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:11.611 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump
2024-05-08T06:44:11.627 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-05-08T06:44:11.628 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
2024-05-08T06:44:11.639 INFO:teuthology.task.internal:Transferring archived files...
2024-05-08T06:44:11.640 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333/remote/smithi031
2024-05-08T06:44:11.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2024-05-08T06:44:11.687 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333/remote/smithi083
2024-05-08T06:44:11.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2024-05-08T06:44:11.720 INFO:teuthology.task.internal:Removing archive directory...
2024-05-08T06:44:11.720 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive
2024-05-08T06:44:11.724 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive
2024-05-08T06:44:11.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload
2024-05-08T06:44:11.782 INFO:teuthology.task.internal:Not uploading archives.
2024-05-08T06:44:11.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.base
2024-05-08T06:44:11.794 INFO:teuthology.task.internal:Tidying up after the test...
2024-05-08T06:44:11.794 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2024-05-08T06:44:11.797 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2024-05-08T06:44:11.810 INFO:teuthology.orchestra.run.smithi031.stdout:   265585      4 drwxr-xr-x   2 ubuntu   ubuntu       4096 May  8 06:44 /home/ubuntu/cephtest
2024-05-08T06:44:11.824 INFO:teuthology.orchestra.run.smithi083.stdout:   262165      4 drwxr-xr-x   2 ubuntu   ubuntu       4096 May  8 06:44 /home/ubuntu/cephtest
2024-05-08T06:44:11.826 DEBUG:teuthology.run_tasks:Unwinding manager kernel
2024-05-08T06:44:11.837 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2024-05-08T06:44:11.861 INFO:teuthology.nuke:Checking targets against current locks
2024-05-08T06:44:11.888 DEBUG:teuthology.nuke:shortname: smithi031
2024-05-08T06:44:11.888 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-05-08T06:44:11.909 DEBUG:teuthology.nuke:shortname: smithi083
2024-05-08T06:44:11.910 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-05-08T06:44:11.917 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 06:31:10.567177', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ/wFSBqluEpbrthgBx8g+89AZVchcHlkbztzGTawEssYvQORjz4qIbOmrca9WYuvIENvE+RwLMjLcbdlPOkDZQ='}
2024-05-08T06:44:11.938 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-05-08_05:24:58-orch:cephadm-wip-adk-testing-2024-05-07-0802-distro-default-smithi/7697333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 06:31:10.569182', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9f/VAz71HOhMPSuDwGIz6HngNnhDL7Cvx7FQ9ZXc9MiDZWWr7IdL2IUPX2bwmxnHCs+sNADkziBblZ19QhZDI='}
2024-05-08T06:44:11.965 INFO:teuthology.orchestra.console.smithi031:Power off
2024-05-08T06:44:11.965 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-05-08T06:44:11.989 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off
2024-05-08T06:44:11.989 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-05-08T06:44:12.005 INFO:teuthology.orchestra.console.smithi083:Power off
2024-05-08T06:44:12.005 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-05-08T06:44:12.014 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on
2024-05-08T06:44:12.027 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off
2024-05-08T06:44:12.027 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-05-08T06:44:12.051 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on
2024-05-08T06:44:16.017 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-05-08T06:44:16.052 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-05-08T06:44:16.128 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on
2024-05-08T06:44:16.162 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on
2024-05-08T06:44:20.130 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-05-08T06:44:20.163 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-05-08T06:44:20.243 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off
2024-05-08T06:44:20.273 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on
2024-05-08T06:44:20.344 INFO:teuthology.orchestra.console.smithi031:Power off completed
2024-05-08T06:44:24.274 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-05-08T06:44:24.384 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off
2024-05-08T06:44:24.484 INFO:teuthology.orchestra.console.smithi083:Power off completed
2024-05-08T06:44:24.589 INFO:teuthology.run:Summary data:
description: orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop
  1-start 2-services/nfs2 3-final}
duration: 256.6727685928345
failure_reason: 'Command failed on smithi031 with status 1: ''sudo /home/ubuntu/cephtest/cephadm
  --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7c8f650b36e258f639fa4a83becade57cbfd2009
  pull'''
owner: scheduled_adking@teuthology
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=744999b3b1aa47cba2ab17deb50e6c47
status: fail
success: false

2024-05-08T06:44:24.589 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-05-08T06:44:24.678 INFO:teuthology.run:FAIL