2023-12-03T21:48:22.331 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-03T21:48:22.331 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_dis@teuthology --archive /home/teuthworker/archive/dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi/7476554 --name dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi --description rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-bitmap 4-supported-random-distro$/{centos_latest} 5-data-pool/none 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} --verbose -- /home/teuthworker/archive/dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi/7476554/orig.config.yaml 2023-12-03T21:48:22.367 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-03T21:48:22.443 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi/7476554 branch: wip-dis-testing-revert description: rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-bitmap 4-supported-random-distro$/{centos_latest} 5-data-pool/none 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} email: idryomov@gmail.com exclude_arch: armv7l first_in_suite: false io_workload: sequential: - qemu: client.0: clone: true disks: - action: none image_name: client.0.0 - action: none image_name: client.0.1 - action: none image_name: client.0.2 test: qa/run_xfstests_qemu.sh type: block job_id: '7476554' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi migrate_workload: sequential: - exec: client.0: - sleep 60 - rbd migration execute client.0.0 - sleep 60 - rbd migration commit client.0.0 - sleep 60 - rbd migration execute client.0.1 name: dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 30000 volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-dis-testing-revert ceph: conf: global: mon warn on pool no app: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-revert sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 owner: scheduled_dis@teuthology priority: 69 repo: https://github.com/ceph/ceph-ci roles: - - mon.a - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mgr.y - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 seed: 2667 sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 sleep_before_teardown: 0 subset: 17/99 suite: rbd suite_branch: wip-dis-testing-revert suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOHMQN33sNUZ4zghkWIGOLOHg72jcbDWVXu6maI8RkOY2IwCRgo2n7+vTytdo//RezS4DOxezv6p7UNXf4u7xko= smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLCQcOYfNXFg+ZfX4nq0CmwD+NLvGXfmYnCHqKf19VQDgQy7a7zAgn3uxthy1dQj+2IqAMAZ3Cjpo2OvY8fONM= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMFkt8WPoautpgXwAnAx78oSFidSXpbmmsOTifhmGVGBAkIMhJbujdaOXtFqvghAq6XHhJdoOi1x1FKlSyglQEY= tasks: - install: null - ceph: null - exec: client.0: - mkdir /home/ubuntu/cephtest/migration - qemu-img create -f qcow2 /home/ubuntu/cephtest/migration/empty.qcow2 1G - echo '{"type":"qcow","stream":{"type":"http","url":"http://download.ceph.com/qa/ubuntu-12.04.qcow2"}}' | rbd migration prepare --import-only --source-spec-path - client.0.0 - rbd migration prepare --import-only --source-spec '{"type":"qcow","stream":{"type":"file","file_path":"/home/ubuntu/cephtest/migration/empty.qcow2"}}' client.0.1 - rbd migration prepare --import-only --source-spec '{"type":"qcow","stream":{"type":"file","file_path":"/home/ubuntu/cephtest/migration/empty.qcow2"}}' client.0.2 - parallel: - io_workload - migrate_workload - exec: client.0: - rm -rf /home/ubuntu/cephtest/migration teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-03_21:41:13 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-03T21:48:22.443 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa; will attempt to use it 2023-12-03T21:48:22.444 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7c730b2996d0b00d56aea5ca791f90c534a13422/qa/tasks 2023-12-03T21:48:22.444 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-03T21:48:22.445 INFO:teuthology.task.internal:Checking packages... 2023-12-03T21:48:22.466 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7c730b2996d0b00d56aea5ca791f90c534a13422' 2023-12-03T21:48:22.466 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-03T21:48:22.466 INFO:teuthology.packaging:ref: None 2023-12-03T21:48:22.466 INFO:teuthology.packaging:tag: None 2023-12-03T21:48:22.466 INFO:teuthology.packaging:branch: wip-dis-testing-revert 2023-12-03T21:48:22.466 INFO:teuthology.packaging:sha1: 7c730b2996d0b00d56aea5ca791f90c534a13422 2023-12-03T21:48:22.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-dis-testing-revert 2023-12-03T21:48:22.605 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7672.g7c730b29 2023-12-03T21:48:22.605 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-03T21:48:22.612 INFO:teuthology.task.internal:no buildpackages task found 2023-12-03T21:48:22.612 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-03T21:48:22.660 INFO:teuthology.task.internal:Saving configuration 2023-12-03T21:48:22.673 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-03T21:48:22.684 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T21:48:22.710 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi/7476554', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 21:42:10.516876', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOHMQN33sNUZ4zghkWIGOLOHg72jcbDWVXu6maI8RkOY2IwCRgo2n7+vTytdo//RezS4DOxezv6p7UNXf4u7xko='} 2023-12-03T21:48:22.735 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi/7476554', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 21:42:10.514572', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLCQcOYfNXFg+ZfX4nq0CmwD+NLvGXfmYnCHqKf19VQDgQy7a7zAgn3uxthy1dQj+2IqAMAZ3Cjpo2OvY8fONM='} 2023-12-03T21:48:22.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi/7476555', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 21:42:10.886166', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMFkt8WPoautpgXwAnAx78oSFidSXpbmmsOTifhmGVGBAkIMhJbujdaOXtFqvghAq6XHhJdoOi1x1FKlSyglQEY='} 2023-12-03T21:48:22.755 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-03T21:48:22.762 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-12-03T21:48:22.762 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-12-03T21:48:22.762 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['client.0'] 2023-12-03T21:48:22.762 INFO:teuthology.run_tasks:Running task console_log... 2023-12-03T21:48:22.873 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f99528679d0>, signals=[15]) 2023-12-03T21:48:22.873 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-03T21:48:22.881 INFO:teuthology.task.internal:Opening connections... 2023-12-03T21:48:22.881 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2023-12-03T21:48:22.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:48:22.982 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2023-12-03T21:48:22.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:48:23.066 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2023-12-03T21:48:23.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:48:23.145 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-03T21:48:23.152 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2023-12-03T21:48:23.170 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2023-12-03T21:48:23.170 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2023-12-03T21:48:23.227 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-03T21:48:23.228 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-03T21:48:23.229 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2023-12-03T21:48:23.254 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2023-12-03T21:48:23.271 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2023-12-03T21:48:23.271 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2023-12-03T21:48:23.328 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-03T21:48:23.329 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-03T21:48:23.330 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2023-12-03T21:48:23.354 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2023-12-03T21:48:23.368 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2023-12-03T21:48:23.368 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2023-12-03T21:48:23.421 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2023-12-03T21:48:23.421 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="9" 2023-12-03T21:48:23.421 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2023-12-03T21:48:23.421 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2023-12-03T21:48:23.421 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="9" 2023-12-03T21:48:23.422 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el9" 2023-12-03T21:48:23.422 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-03T21:48:23.422 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2023-12-03T21:48:23.422 INFO:teuthology.orchestra.run.smithi167.stdout:LOGO="fedora-logo-icon" 2023-12-03T21:48:23.422 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-03T21:48:23.422 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2023-12-03T21:48:23.422 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-03T21:48:23.422 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-03T21:48:23.422 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-03T21:48:23.423 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2023-12-03T21:48:23.437 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-03T21:48:23.446 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-03T21:48:23.469 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-03T21:48:23.469 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2023-12-03T21:48:23.472 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2023-12-03T21:48:23.474 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2023-12-03T21:48:23.488 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-03T21:48:23.494 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-03T21:48:23.495 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2023-12-03T21:48:23.529 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2023-12-03T21:48:23.532 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2023-12-03T21:48:23.635 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-03T21:48:23.710 INFO:teuthology.run_tasks:Running task kernel... 2023-12-03T21:48:23.725 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-03T21:48:23.726 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-03T21:48:23.726 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-03T21:48:23.726 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-03T21:48:23.727 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-03T21:48:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T21:48:23.741 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2023-12-03T21:48:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T21:48:23.743 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2023-12-03T21:48:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T21:48:23.744 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-12-03T21:48:23.796 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-361.el9.x86_64 2023-12-03T21:48:23.796 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.14.0-361.el9.x86_64 2023-12-03T21:48:23.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2023-12-03T21:48:23.800 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-361.el9.x86_64 2023-12-03T21:48:23.800 INFO:teuthology.task.kernel:Running kernel on smithi076: 5.14.0-361.el9.x86_64 2023-12-03T21:48:23.800 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2023-12-03T21:48:23.801 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-361.el9.x86_64 2023-12-03T21:48:23.801 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.14.0-361.el9.x86_64 2023-12-03T21:48:23.801 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2023-12-03T21:48:24.014 INFO:teuthology.orchestra.run.smithi167.stdout:Waiting for process with pid 5553 to finish. 2023-12-03T21:48:26.458 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - AppStream 6.6 MB/s | 6.4 MB 00:00 2023-12-03T21:48:26.467 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 6.7 MB/s | 6.4 MB 00:00 2023-12-03T21:48:28.133 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2023-12-03T21:48:28.141 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2023-12-03T21:48:29.082 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 7.9 MB 00:00 2023-12-03T21:48:29.332 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 7.9 MB 00:00 2023-12-03T21:48:31.366 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 21 MB/s | 18 MB 00:00 2023-12-03T21:48:32.052 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 18 MB 00:01 2023-12-03T21:48:33.422 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:02 ago on Sun 03 Dec 2023 09:48:31 PM UTC. 2023-12-03T21:48:34.403 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-03T21:48:34.403 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-03T21:48:34.471 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2023-12-03T21:48:34.480 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2023-12-03T21:48:34.480 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2023-12-03T21:48:34.480 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2023-12-03T21:48:34.480 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2023-12-03T21:48:34.480 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-03T21:48:34.480 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2023-12-03T21:48:34.480 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-03T21:48:34.481 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-03T21:48:34.481 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-03T21:48:34.481 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-12-03T21:48:34.481 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2023-12-03T21:48:34.481 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2023-12-03T21:48:34.481 INFO:teuthology.orchestra.run.smithi167.stdout:Install 4 Packages 2023-12-03T21:48:34.481 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-12-03T21:48:34.482 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 96 M 2023-12-03T21:48:34.482 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 121 M 2023-12-03T21:48:34.482 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2023-12-03T21:48:35.167 INFO:teuthology.orchestra.run.smithi167.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 10 MB/s | 5.6 MB 00:00 2023-12-03T21:48:35.709 INFO:teuthology.orchestra.run.smithi167.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 18 MB/s | 20 MB 00:01 2023-12-03T21:48:35.725 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 5.8 MB 00:00 2023-12-03T21:48:35.942 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.8 MB 00:00 2023-12-03T21:48:36.784 INFO:teuthology.orchestra.run.smithi167.stdout:(3/4): kernel-modules-core-5.14.0-388.el9.x86_6 20 MB/s | 32 MB 00:01 2023-12-03T21:48:37.059 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 15 kB 00:00 2023-12-03T21:48:37.202 INFO:teuthology.orchestra.run.smithi167.stdout:(4/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 15 MB/s | 38 MB 00:02 2023-12-03T21:48:37.202 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2023-12-03T21:48:37.202 INFO:teuthology.orchestra.run.smithi167.stdout:Total 35 MB/s | 96 MB 00:02 2023-12-03T21:48:37.486 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2023-12-03T21:48:37.997 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2023-12-03T21:48:38.080 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2023-12-03T21:48:38.456 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2023-12-03T21:48:38.456 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2023-12-03T21:48:39.737 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 20 MB 00:02 2023-12-03T21:48:39.749 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2023-12-03T21:48:39.750 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2023-12-03T21:48:42.699 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2023-12-03T21:48:42.927 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2023-12-03T21:48:43.348 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-03T21:48:43.469 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T21:48:44.495 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-12-03T21:48:45.317 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-03T21:48:45.317 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-03T21:48:45.416 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-03T21:48:45.422 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-03T21:48:45.423 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-03T21:48:45.423 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-03T21:48:45.423 INFO:teuthology.orchestra.run.smithi033.stdout:Install 4 Packages 2023-12-03T21:48:45.423 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-03T21:48:45.423 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 96 M 2023-12-03T21:48:45.423 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 121 M 2023-12-03T21:48:45.423 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-03T21:48:46.243 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T21:48:46.447 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T21:48:46.835 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-03T21:48:46.835 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-03T21:48:46.928 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-03T21:48:46.933 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-03T21:48:46.934 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-03T21:48:46.935 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-03T21:48:46.935 INFO:teuthology.orchestra.run.smithi076.stdout:Install 4 Packages 2023-12-03T21:48:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-03T21:48:46.935 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 96 M 2023-12-03T21:48:46.935 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 121 M 2023-12-03T21:48:46.935 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-03T21:48:47.762 INFO:teuthology.orchestra.run.smithi033.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 2.5 MB/s | 5.6 MB 00:02 2023-12-03T21:48:47.897 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 6.1 MB/s | 5.6 MB 00:00 2023-12-03T21:48:48.439 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 14 MB/s | 20 MB 00:01 2023-12-03T21:48:50.053 INFO:teuthology.orchestra.run.smithi033.stdout:(2/4): kernel-modules-core-5.14.0-388.el9.x86_6 14 MB/s | 32 MB 00:02 2023-12-03T21:48:50.423 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): kernel-modules-core-5.14.0-388.el9.x86_6 13 MB/s | 32 MB 00:02 2023-12-03T21:48:50.798 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 9.9 MB/s | 38 MB 00:03 2023-12-03T21:48:50.798 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-03T21:48:50.798 INFO:teuthology.orchestra.run.smithi076.stdout:Total 25 MB/s | 96 MB 00:03 2023-12-03T21:48:51.082 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T21:48:51.289 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:48:51.657 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-03T21:48:52.037 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-03T21:48:52.037 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-03T21:48:53.657 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-03T21:48:53.657 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-03T21:48:55.891 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:48:55.961 INFO:teuthology.orchestra.run.smithi033.stdout:(3/4): kernel-core-5.14.0-388.el9.x86_64.rpm 1.9 MB/s | 20 MB 00:10 2023-12-03T21:48:56.593 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-03T21:48:57.244 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-03T21:48:57.369 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T21:49:00.119 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T21:49:00.302 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T21:49:04.452 INFO:teuthology.orchestra.run.smithi033.stdout:(4/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 2.0 MB/s | 38 MB 00:18 2023-12-03T21:49:04.452 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-03T21:49:04.452 INFO:teuthology.orchestra.run.smithi033.stdout:Total 5.0 MB/s | 96 MB 00:19 2023-12-03T21:49:04.936 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T21:49:05.068 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:05.343 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-03T21:49:05.770 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-03T21:49:05.770 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-03T21:49:07.150 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-03T21:49:07.150 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-03T21:49:09.742 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:09.922 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-03T21:49:10.617 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-03T21:49:10.760 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T21:49:13.666 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T21:49:13.846 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T21:49:18.802 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T21:49:18.922 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:23.909 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:24.981 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:25.024 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:25.394 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:25.394 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-03T21:49:25.394 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T21:49:25.394 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T21:49:25.804 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:25.804 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-12-03T21:49:25.804 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2023-12-03T21:49:25.804 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-03T21:49:25.804 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-03T21:49:25.804 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-03T21:49:25.804 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-03T21:49:25.804 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-12-03T21:49:25.804 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2023-12-03T21:49:26.263 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T21:49:26.405 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T21:49:26.405 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T21:49:26.406 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-03T21:49:26.406 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-03T21:49:26.406 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-03T21:49:26.406 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-12-03T21:49:26.406 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2023-12-03T21:49:26.406 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T21:49:26.526 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T21:49:26.527 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T21:49:26.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo rpm -qi grub2-tools 2023-12-03T21:49:26.561 INFO:teuthology.orchestra.run.smithi167.stdout:Name : grub2-tools 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Epoch : 1 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Version : 2.06 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Release : 68.el9 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: x86_64 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Group : Unspecified 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Size : 8264254 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:License : GPLv3+ 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-03T21:49:26.562 INFO:teuthology.orchestra.run.smithi167.stdout:Packager : builder@centos.org 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout:Vendor : CentOS 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout:Summary : Support tools for GRUB. 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout:Description : 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout:hardware devices. 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-12-03T21:49:26.563 INFO:teuthology.orchestra.run.smithi167.stdout:This subpackage provides tools for support of all platforms. 2023-12-03T21:49:26.564 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-03T21:49:26.564 INFO:teuthology.task.kernel:Updating grub on smithi167 to boot 5.14.0-388.el9.x86_64 2023-12-03T21:49:26.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-03T21:49:27.180 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2023-12-03T21:49:28.640 INFO:teuthology.orchestra.run.smithi167.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-03T21:49:28.677 INFO:teuthology.orchestra.run.smithi167.stderr:done 2023-12-03T21:49:28.679 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2023-12-03T21:49:28.693 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.kbL4KoBaol 2023-12-03T21:49:28.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kbL4KoBaol 2023-12-03T21:49:28.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.kbL4KoBaol 2023-12-03T21:49:28.909 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.kbL4KoBaol is 6KB 2023-12-03T21:49:28.929 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.kbL4KoBaol 2023-12-03T21:49:28.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo /bin/ls /boot/loader/entries || true 2023-12-03T21:49:29.015 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-03T21:49:29.015 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-03T21:49:29.015 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-03T21:49:29.015 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-03T21:49:29.016 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-03T21:49:29.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2023-12-03T21:49:38.591 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:38.633 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:39.033 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:39.033 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-03T21:49:39.033 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T21:49:39.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T21:49:39.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:39.439 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-03T21:49:39.439 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-03T21:49:39.439 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-03T21:49:39.439 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-03T21:49:39.439 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-03T21:49:39.439 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-03T21:49:39.439 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-03T21:49:39.439 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-03T21:49:39.979 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T21:49:40.129 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T21:49:40.129 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T21:49:40.129 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-03T21:49:40.129 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-03T21:49:40.129 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-03T21:49:40.129 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-03T21:49:40.129 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2023-12-03T21:49:40.130 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T21:49:40.247 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T21:49:40.248 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T21:49:40.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2023-12-03T21:49:40.281 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.06 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 68.el9 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Group : Unspecified 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 8264254 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-03T21:49:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : builder@centos.org 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2023-12-03T21:49:40.283 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-03T21:49:40.284 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2023-12-03T21:49:40.284 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-03T21:49:40.284 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 5.14.0-388.el9.x86_64 2023-12-03T21:49:40.285 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-03T21:49:40.911 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2023-12-03T21:49:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-03T21:49:42.340 INFO:teuthology.orchestra.run.smithi076.stderr:done 2023-12-03T21:49:42.342 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2023-12-03T21:49:42.357 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.08P8wg19vv 2023-12-03T21:49:42.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.08P8wg19vv 2023-12-03T21:49:42.426 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.08P8wg19vv 2023-12-03T21:49:42.562 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.08P8wg19vv is 6KB 2023-12-03T21:49:42.611 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.08P8wg19vv 2023-12-03T21:49:42.625 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2023-12-03T21:49:42.704 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-03T21:49:42.704 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-03T21:49:42.704 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-03T21:49:42.704 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-03T21:49:42.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-03T21:49:42.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2023-12-03T21:49:54.226 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:54.272 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:54.739 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:54.739 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-03T21:49:54.739 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T21:49:54.739 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T21:49:55.171 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T21:49:55.171 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-03T21:49:55.171 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-03T21:49:55.171 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-03T21:49:55.171 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-03T21:49:55.171 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-03T21:49:55.171 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-03T21:49:55.171 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-03T21:49:55.171 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-03T21:49:55.657 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T21:49:55.803 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T21:49:55.804 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T21:49:55.804 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-03T21:49:55.804 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-03T21:49:55.804 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-03T21:49:55.804 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-03T21:49:55.804 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2023-12-03T21:49:55.804 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T21:49:55.927 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T21:49:55.927 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T21:49:55.928 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2023-12-03T21:49:55.965 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2023-12-03T21:49:55.965 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 68.el9 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8264254 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2023-12-03T21:49:55.966 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-03T21:49:55.967 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2023-12-03T21:49:55.968 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-03T21:49:55.968 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 5.14.0-388.el9.x86_64 2023-12-03T21:49:55.968 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-03T21:49:56.566 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2023-12-03T21:49:57.914 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-03T21:49:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:done 2023-12-03T21:49:57.961 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-12-03T21:49:57.977 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.dJpEg6w9os 2023-12-03T21:49:57.977 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dJpEg6w9os 2023-12-03T21:49:58.050 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.dJpEg6w9os 2023-12-03T21:49:58.194 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.dJpEg6w9os is 6KB 2023-12-03T21:49:58.242 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.dJpEg6w9os 2023-12-03T21:49:58.257 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2023-12-03T21:49:58.326 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-03T21:49:58.326 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-03T21:49:58.326 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-03T21:49:58.326 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-03T21:49:58.327 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-03T21:49:58.475 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2023-12-03T21:49:59.150 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-12-03T21:49:59.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:49:59.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:50:12.874 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-03T21:50:12.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-03T21:50:12.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:50:17.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:50:26.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:50:26.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:50:28.479 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-03T21:50:28.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-03T21:50:28.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:50:29.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:50:31.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2023-12-03T21:50:40.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-03T21:50:40.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:50:41.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:50:41.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:50:43.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2023-12-03T21:50:44.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:50:46.882 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-12-03T21:50:55.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-03T21:50:55.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:50:55.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-03T21:50:55.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:50:58.402 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2023-12-03T21:50:59.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:50:59.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:51:02.914 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:51:13.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-03T21:51:13.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:51:14.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-12-03T21:51:20.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:51:20.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:51:24.002 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:51:26.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-03T21:51:26.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:51:27.009 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-12-03T21:51:27.468 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-03T21:51:27.468 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-03T21:51:27.468 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-12-03T21:51:27.485 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-388.el9.x86_64 2023-12-03T21:51:27.485 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-03T21:51:27.486 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-03T21:51:27.486 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-03T21:51:28.487 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-03T21:51:28.487 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-03T21:51:28.577 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2023-12-03T21:51:28.618 DEBUG:teuthology.parallel:result is None 2023-12-03T21:51:45.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:51:45.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:51:45.289 DEBUG:teuthology.orchestra.run.smithi076:> true 2023-12-03T21:51:45.627 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-03T21:51:45.627 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-388.el9.x86_64"... 2023-12-03T21:51:45.628 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2023-12-03T21:51:45.642 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-388.el9.x86_64 2023-12-03T21:51:45.642 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-03T21:51:45.642 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-03T21:51:45.642 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-03T21:51:46.643 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-03T21:51:46.643 DEBUG:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-03T21:51:46.695 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2023-12-03T21:51:46.740 DEBUG:teuthology.parallel:result is None 2023-12-03T21:51:48.066 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:52:12.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:52:12.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:52:15.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:52:42.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:52:42.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:52:45.218 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:53:15.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:53:15.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:53:18.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:53:51.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:53:51.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:53:54.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:54:30.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:54:30.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:54:33.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:55:12.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:55:12.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:55:15.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:55:45.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:55:45.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:55:48.546 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:55:48.546 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (14) after waiting for 300 seconds 2023-12-03T21:55:48.547 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-388.el9.x86_64"... 2023-12-03T21:55:48.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:55:48.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:55:51.618 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:56:00.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:56:00.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:56:03.682 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:56:15.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:56:15.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:56:18.754 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:56:21.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-12-03T21:56:21.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T21:56:21.826 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-12-03T21:56:21.826 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-03T21:56:21.826 ERROR:teuthology.task.kernel:Saw exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 710, in wait_for_reboot assert not need_to_install(ctx, client, need_install[client]), \ File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 182, in need_to_install ctx.cluster.only(role).run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi167 2023-12-03T21:56:21.828 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 1237, in task p.spawn(process_role, ctx, config, timeout, role, role_config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 1342, in process_role wait_for_reboot(ctx, {role: need_version}, timeout, config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 710, in wait_for_reboot assert not need_to_install(ctx, client, need_install[client]), \ File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 182, in need_to_install ctx.cluster.only(role).run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi167 2023-12-03T21:56:21.904 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c131491119784bb889001016d202c27a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 1237, in task p.spawn(process_role, ctx, config, timeout, role, role_config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 1342, in process_role wait_for_reboot(ctx, {role: need_version}, timeout, config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 710, in wait_for_reboot assert not need_to_install(ctx, client, need_install[client]), \ File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/kernel.py", line 182, in need_to_install ctx.cluster.only(role).run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi167 2023-12-03T21:56:21.907 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-03T21:56:21.915 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-03T21:56:22.003 INFO:teuthology.nuke:Checking targets against current locks 2023-12-03T21:56:22.047 DEBUG:teuthology.nuke:shortname: smithi033 2023-12-03T21:56:22.048 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T21:56:22.066 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi/7476554', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 21:42:10.516876', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOHMQN33sNUZ4zghkWIGOLOHg72jcbDWVXu6maI8RkOY2IwCRgo2n7+vTytdo//RezS4DOxezv6p7UNXf4u7xko='} 2023-12-03T21:56:22.070 DEBUG:teuthology.nuke:shortname: smithi076 2023-12-03T21:56:22.070 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T21:56:22.091 INFO:teuthology.nuke:Not nuking smithi167.front.sepia.ceph.com because description doesn't match: dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi != None 2023-12-03T21:56:22.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-03_21:41:13-rbd-wip-dis-testing-revert-distro-default-smithi/7476554', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 21:42:10.514572', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCLCQcOYfNXFg+ZfX4nq0CmwD+NLvGXfmYnCHqKf19VQDgQy7a7zAgn3uxthy1dQj+2IqAMAZ3Cjpo2OvY8fONM='} 2023-12-03T21:56:22.126 INFO:teuthology.orchestra.console.smithi033:Power off 2023-12-03T21:56:22.127 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-03T21:56:22.154 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2023-12-03T21:56:22.154 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T21:56:22.168 INFO:teuthology.orchestra.console.smithi076:Power off 2023-12-03T21:56:22.168 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-03T21:56:22.178 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-12-03T21:56:22.190 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2023-12-03T21:56:22.190 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T21:56:22.213 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2023-12-03T21:56:26.179 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T21:56:26.214 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T21:56:26.291 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-12-03T21:56:26.324 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2023-12-03T21:56:30.294 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T21:56:30.325 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T21:56:30.404 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-12-03T21:56:30.435 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2023-12-03T21:56:30.535 INFO:teuthology.orchestra.console.smithi076:Power off completed 2023-12-03T21:56:34.406 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T21:56:34.516 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2023-12-03T21:56:34.617 INFO:teuthology.orchestra.console.smithi033:Power off completed 2023-12-03T21:56:34.719 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMFkt8WPoautpgXwAnAx78oSFidSXpbmmsOTifhmGVGBAkIMhJbujdaOXtFqvghAq6XHhJdoOi1x1FKlSyglQEY= 2023-12-03T21:56:35.561 INFO:teuthology.run:Summary data: description: rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-bitmap 4-supported-random-distro$/{centos_latest} 5-data-pool/none 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} failure_reason: Failed to reconnect to smithi167 owner: scheduled_dis@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c131491119784bb889001016d202c27a status: fail success: false 2023-12-03T21:56:35.561 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-03T21:56:35.649 INFO:teuthology.run:FAIL