2023-10-18T14:26:02.296 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-18T14:26:02.298 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543 --description smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/kclient_workunit_suites_pjd}} --name yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543/orig.config.yaml 2023-10-18T14:26:02.335 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:26:02.415 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543 branch: quincy-release description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/kclient_workunit_suites_pjd}} email: null first_in_suite: false job_id: '7431543' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 2 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 10 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: quincy-release ceph: conf: global: ms die on skipped message: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-quincy-test-fixes sha1: 69fb3d4792adb6f1b63824c294e62c636f9a850d owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mds.a - mgr.x - osd.0 - osd.1 - - mon.b - mds.b - mon.c - mgr.y - osd.2 - osd.3 - - client.0 seed: 7103 sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 sleep_before_teardown: 0 subset: 111/120000 suite: smoke suite_branch: wip-quincy-test-fixes suite_path: /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa suite_relpath: qa suite_repo: https://github.com/pdvian/ceph.git suite_sha1: 69fb3d4792adb6f1b63824c294e62c636f9a850d targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEyd4ZBBCFjtHi1KpvJEZ6aUvCCopLOk4ky3JtJ17F6ij9MbiUoePsyXLJ4i+2rGvHfAfA4PUF+8adBozY6Aeko= smithi158.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFrqcoDjUYn/LXidl8axWJyRs+XJ3/uOtDLNmfZRp34oFMKFpEAfTtLY6wiYC+w0YqEqucQATNt7n0tH4uHGYWc= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDtqD8ljRQetAY1Eb+0EyWFlHbkHF68fptbIqoJsT5LmtRbYpiBMe6owPxU6npT8RnN4tKmoBV5W2apfD19b8Yc= tasks: - install: cleanup: true - ceph: fs: xfs log-ignorelist: - \(POOL_APP_NOT_ENABLED\) - kclient: null - workunit: clients: all: - suites/pjd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-18_13:55:44 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T14:26:02.416 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa; will attempt to use it 2023-10-18T14:26:02.417 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_pdvian_ceph_69fb3d4792adb6f1b63824c294e62c636f9a850d/qa/tasks 2023-10-18T14:26:02.417 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T14:26:02.419 INFO:teuthology.task.internal:Checking packages... 2023-10-18T14:26:02.445 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '9d6d32bb3452d5179bde2ee1cfa05df7a65f4586' 2023-10-18T14:26:02.445 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T14:26:02.446 INFO:teuthology.packaging:ref: None 2023-10-18T14:26:02.446 INFO:teuthology.packaging:tag: None 2023-10-18T14:26:02.446 INFO:teuthology.packaging:branch: quincy-release 2023-10-18T14:26:02.447 INFO:teuthology.packaging:sha1: 9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 2023-10-18T14:26:02.447 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy-release 2023-10-18T14:26:02.687 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1415.g9d6d32bb 2023-10-18T14:26:02.688 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T14:26:02.694 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T14:26:02.694 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T14:26:02.716 INFO:teuthology.task.internal:Saving configuration 2023-10-18T14:26:02.729 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T14:26:02.735 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:26:02.761 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:14.055551', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEyd4ZBBCFjtHi1KpvJEZ6aUvCCopLOk4ky3JtJ17F6ij9MbiUoePsyXLJ4i+2rGvHfAfA4PUF+8adBozY6Aeko='} 2023-10-18T14:26:02.786 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:14.059010', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFrqcoDjUYn/LXidl8axWJyRs+XJ3/uOtDLNmfZRp34oFMKFpEAfTtLY6wiYC+w0YqEqucQATNt7n0tH4uHGYWc='} 2023-10-18T14:26:02.810 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:14.057425', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDtqD8ljRQetAY1Eb+0EyWFlHbkHF68fptbIqoJsT5LmtRbYpiBMe6owPxU6npT8RnN4tKmoBV5W2apfD19b8Yc='} 2023-10-18T14:26:02.810 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T14:26:02.817 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-10-18T14:26:02.818 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-10-18T14:26:02.818 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['client.0'] 2023-10-18T14:26:02.819 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T14:26:02.969 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8a28efda60>, signals=[15]) 2023-10-18T14:26:02.970 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T14:26:02.976 INFO:teuthology.task.internal:Opening connections... 2023-10-18T14:26:02.977 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2023-10-18T14:26:02.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:03.045 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi158.front.sepia.ceph.com 2023-10-18T14:26:03.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:03.112 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2023-10-18T14:26:03.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:03.183 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T14:26:03.190 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2023-10-18T14:26:03.210 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2023-10-18T14:26:03.211 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2023-10-18T14:26:03.267 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:26:03.267 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T14:26:03.267 INFO:teuthology.orchestra.run.smithi114.stdout:ID="rhel" 2023-10-18T14:26:03.267 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="fedora" 2023-10-18T14:26:03.268 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8.4" 2023-10-18T14:26:03.268 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:26:03.268 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T14:26:03.268 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2023-10-18T14:26:03.268 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T14:26:03.269 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:26:03.269 INFO:teuthology.orchestra.run.smithi114.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:26:03.269 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:26:03.269 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-18T14:26:03.269 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:26:03.271 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T14:26:03.271 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:26:03.271 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T14:26:03.272 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2023-10-18T14:26:03.293 DEBUG:teuthology.orchestra.run.smithi158:> uname -m 2023-10-18T14:26:03.310 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2023-10-18T14:26:03.311 DEBUG:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2023-10-18T14:26:03.367 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:26:03.367 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T14:26:03.368 INFO:teuthology.orchestra.run.smithi158.stdout:ID="rhel" 2023-10-18T14:26:03.368 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="fedora" 2023-10-18T14:26:03.368 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="8.4" 2023-10-18T14:26:03.368 INFO:teuthology.orchestra.run.smithi158.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:26:03.369 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T14:26:03.369 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31" 2023-10-18T14:26:03.369 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T14:26:03.369 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:26:03.369 INFO:teuthology.orchestra.run.smithi158.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:26:03.370 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:26:03.370 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-18T14:26:03.370 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:26:03.370 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T14:26:03.370 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:26:03.371 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T14:26:03.371 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2023-10-18T14:26:03.392 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2023-10-18T14:26:03.410 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2023-10-18T14:26:03.411 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2023-10-18T14:26:03.466 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:26:03.467 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T14:26:03.467 INFO:teuthology.orchestra.run.smithi172.stdout:ID="rhel" 2023-10-18T14:26:03.467 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="fedora" 2023-10-18T14:26:03.467 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8.4" 2023-10-18T14:26:03.468 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:26:03.468 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T14:26:03.468 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2023-10-18T14:26:03.468 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T14:26:03.468 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:26:03.469 INFO:teuthology.orchestra.run.smithi172.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:26:03.469 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:26:03.469 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-18T14:26:03.469 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:26:03.469 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T14:26:03.470 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:26:03.470 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T14:26:03.470 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2023-10-18T14:26:03.492 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T14:26:03.502 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T14:26:03.508 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T14:26:03.508 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:26:03.511 DEBUG:teuthology.orchestra.run.smithi158:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:26:03.513 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:26:03.528 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T14:26:03.537 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T14:26:03.538 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:26:03.567 DEBUG:teuthology.orchestra.run.smithi158:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:26:03.569 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:26:03.581 INFO:teuthology.orchestra.run.smithi114.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T14:26:03.583 INFO:teuthology.orchestra.run.smithi158.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T14:26:03.584 INFO:teuthology.orchestra.run.smithi172.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T14:26:03.585 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T14:26:03.653 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T14:26:03.707 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-18T14:26:03.708 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-18T14:26:03.708 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:26:03.709 DEBUG:teuthology.orchestra.run.smithi158:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:26:03.710 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:26:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:26:03.725 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-10-18T14:26:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:26:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:26:03.727 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2023-10-18T14:26:03.728 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2023-10-18T14:26:03.781 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:03.781 INFO:teuthology.task.kernel:Running kernel on smithi172: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:03.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2023-10-18T14:26:03.783 INFO:teuthology.orchestra.run.smithi158.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:03.783 INFO:teuthology.task.kernel:Running kernel on smithi158: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:03.784 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum install -y kernel 2023-10-18T14:26:03.785 INFO:teuthology.orchestra.run.smithi114.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:03.785 INFO:teuthology.task.kernel:Running kernel on smithi114: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T14:26:03.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2023-10-18T14:26:05.202 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:05.203 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-10-18T14:26:05.249 INFO:teuthology.orchestra.run.smithi114.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:05.250 INFO:teuthology.orchestra.run.smithi114.stdout:Unable to read consumer identity 2023-10-18T14:26:05.277 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:05.278 INFO:teuthology.orchestra.run.smithi158.stdout:Unable to read consumer identity 2023-10-18T14:26:06.649 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-10-18T14:26:06.668 INFO:teuthology.orchestra.run.smithi114.stdout:Copr repo for python3-asyncssh owned by ceph 35 kB/s | 3.5 kB 00:00 2023-10-18T14:26:06.792 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2023-10-18T14:26:07.618 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-10-18T14:26:07.644 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-10-18T14:26:08.010 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-10-18T14:26:10.914 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2023-10-18T14:26:10.936 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 4.0 MB/s | 24 kB 00:00 2023-10-18T14:26:11.290 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-10-18T14:26:11.666 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T14:26:11.666 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T14:26:11.666 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T14:26:11.669 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T14:26:11.670 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T14:26:11.670 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T14:26:11.812 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2023-10-18T14:26:11.813 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2023-10-18T14:26:11.813 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2023-10-18T14:26:11.827 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-18T14:26:11.828 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-18T14:26:11.828 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-18T14:26:11.936 DEBUG:teuthology.orchestra.run.smithi172:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:26:11.938 DEBUG:teuthology.orchestra.run.smithi114:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:26:12.027 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T14:26:12.027 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T14:26:12.028 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T14:26:12.186 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2023-10-18T14:26:12.187 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2023-10-18T14:26:12.187 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2023-10-18T14:26:12.280 INFO:teuthology.orchestra.run.smithi114.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:12.281 INFO:teuthology.orchestra.run.smithi114.stdout:Unable to read consumer identity 2023-10-18T14:26:12.283 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:12.283 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-10-18T14:26:12.292 DEBUG:teuthology.orchestra.run.smithi158:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:26:12.576 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:26:10 PM UTC. 2023-10-18T14:26:12.579 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:26:10 PM UTC. 2023-10-18T14:26:12.678 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:12.678 INFO:teuthology.orchestra.run.smithi158.stdout:Unable to read consumer identity 2023-10-18T14:26:12.725 INFO:teuthology.orchestra.run.smithi114.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:12.726 INFO:teuthology.orchestra.run.smithi114.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:12.735 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:12.736 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:12.736 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:12.752 INFO:teuthology.orchestra.run.smithi114.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:12.765 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:12.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum reinstall -y kernel || true 2023-10-18T14:26:12.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum reinstall -y kernel || true 2023-10-18T14:26:12.976 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:00:01 ago on Wed 18 Oct 2023 02:26:11 PM UTC. 2023-10-18T14:26:13.122 INFO:teuthology.orchestra.run.smithi158.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:13.122 INFO:teuthology.orchestra.run.smithi158.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:13.123 INFO:teuthology.orchestra.run.smithi158.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:13.144 INFO:teuthology.orchestra.run.smithi114.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:13.144 INFO:teuthology.orchestra.run.smithi114.stdout:Unable to read consumer identity 2023-10-18T14:26:13.146 INFO:teuthology.orchestra.run.smithi158.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:13.169 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:13.169 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-10-18T14:26:13.196 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum reinstall -y kernel || true 2023-10-18T14:26:13.433 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:03 ago on Wed 18 Oct 2023 02:26:10 PM UTC. 2023-10-18T14:26:13.457 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:03 ago on Wed 18 Oct 2023 02:26:10 PM UTC. 2023-10-18T14:26:13.541 INFO:teuthology.orchestra.run.smithi158.stdout:Updating Subscription Management repositories. 2023-10-18T14:26:13.541 INFO:teuthology.orchestra.run.smithi158.stdout:Unable to read consumer identity 2023-10-18T14:26:13.574 INFO:teuthology.orchestra.run.smithi114.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:13.575 INFO:teuthology.orchestra.run.smithi114.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:13.575 INFO:teuthology.orchestra.run.smithi114.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:13.576 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:13.576 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:13.577 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:13.589 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:13.589 INFO:teuthology.orchestra.run.smithi114.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:13.638 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:13.644 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:13.763 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.763 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.763 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.764 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:26:13.764 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:26:13.764 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-18T14:26:13.764 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:26:13.765 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:13.770 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.770 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.771 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.771 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:26:13.771 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:26:13.771 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-18T14:26:13.772 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:26:13.772 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:13.836 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:26:11 PM UTC. 2023-10-18T14:26:13.889 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.889 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2023-10-18T14:26:13.896 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.897 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2023-10-18T14:26:13.916 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2023-10-18T14:26:13.917 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2023-10-18T14:26:13.917 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.02 2023-10-18T14:26:13.917 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 99.el8_4.1 2023-10-18T14:26:13.917 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2023-10-18T14:26:13.917 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T14:26:13.918 INFO:teuthology.orchestra.run.smithi114.stdout:Group : System Environment/Base 2023-10-18T14:26:13.918 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 9264631 2023-10-18T14:26:13.918 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2023-10-18T14:26:13.918 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T14:26:13.918 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T14:26:13.919 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T14:26:13.919 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T14:26:13.919 INFO:teuthology.orchestra.run.smithi114.stdout:Relocations : (not relocatable) 2023-10-18T14:26:13.919 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : Red Hat, Inc. 2023-10-18T14:26:13.919 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : Red Hat, Inc. 2023-10-18T14:26:13.919 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:26:13.920 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2023-10-18T14:26:13.920 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2023-10-18T14:26:13.920 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-18T14:26:13.920 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:26:13.920 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:26:13.921 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:26:13.921 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2023-10-18T14:26:13.921 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-18T14:26:13.921 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:26:13.922 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:26:13.922 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:26:13.924 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2023-10-18T14:26:13.925 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2023-10-18T14:26:13.925 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.02 2023-10-18T14:26:13.925 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 99.el8_4.1 2023-10-18T14:26:13.925 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2023-10-18T14:26:13.925 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T14:26:13.926 INFO:teuthology.orchestra.run.smithi172.stdout:Group : System Environment/Base 2023-10-18T14:26:13.926 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 9264631 2023-10-18T14:26:13.926 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2023-10-18T14:26:13.926 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T14:26:13.926 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T14:26:13.927 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T14:26:13.927 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T14:26:13.927 INFO:teuthology.orchestra.run.smithi172.stdout:Relocations : (not relocatable) 2023-10-18T14:26:13.927 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : Red Hat, Inc. 2023-10-18T14:26:13.927 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : Red Hat, Inc. 2023-10-18T14:26:13.928 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:26:13.928 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2023-10-18T14:26:13.928 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2023-10-18T14:26:13.928 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-18T14:26:13.928 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:26:13.929 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:26:13.929 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:26:13.929 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2023-10-18T14:26:13.929 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-18T14:26:13.929 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:26:13.930 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:26:13.930 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:13.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:26:13.971 INFO:teuthology.orchestra.run.smithi158.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T14:26:13.971 INFO:teuthology.orchestra.run.smithi158.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:13.971 INFO:teuthology.orchestra.run.smithi158.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T14:26:13.979 INFO:teuthology.orchestra.run.smithi158.stderr:Error: No packages marked for reinstall. 2023-10-18T14:26:14.030 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:14.155 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:14.155 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:14.155 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:14.156 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:26:14.156 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:26:14.156 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-18T14:26:14.156 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi158.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:26:14.157 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:26:14.282 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:14.282 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:14.283 DEBUG:teuthology.orchestra.run.smithi158:> sudo rpm -qi grub2-tools 2023-10-18T14:26:14.310 INFO:teuthology.orchestra.run.smithi158.stdout:Name : grub2-tools 2023-10-18T14:26:14.310 INFO:teuthology.orchestra.run.smithi158.stdout:Epoch : 1 2023-10-18T14:26:14.310 INFO:teuthology.orchestra.run.smithi158.stdout:Version : 2.02 2023-10-18T14:26:14.310 INFO:teuthology.orchestra.run.smithi158.stdout:Release : 99.el8_4.1 2023-10-18T14:26:14.311 INFO:teuthology.orchestra.run.smithi158.stdout:Architecture: x86_64 2023-10-18T14:26:14.311 INFO:teuthology.orchestra.run.smithi158.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T14:26:14.311 INFO:teuthology.orchestra.run.smithi158.stdout:Group : System Environment/Base 2023-10-18T14:26:14.311 INFO:teuthology.orchestra.run.smithi158.stdout:Size : 9264631 2023-10-18T14:26:14.311 INFO:teuthology.orchestra.run.smithi158.stdout:License : GPLv3+ 2023-10-18T14:26:14.312 INFO:teuthology.orchestra.run.smithi158.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T14:26:14.312 INFO:teuthology.orchestra.run.smithi158.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T14:26:14.312 INFO:teuthology.orchestra.run.smithi158.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T14:26:14.312 INFO:teuthology.orchestra.run.smithi158.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T14:26:14.312 INFO:teuthology.orchestra.run.smithi158.stdout:Relocations : (not relocatable) 2023-10-18T14:26:14.313 INFO:teuthology.orchestra.run.smithi158.stdout:Packager : Red Hat, Inc. 2023-10-18T14:26:14.313 INFO:teuthology.orchestra.run.smithi158.stdout:Vendor : Red Hat, Inc. 2023-10-18T14:26:14.313 INFO:teuthology.orchestra.run.smithi158.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:26:14.313 INFO:teuthology.orchestra.run.smithi158.stdout:Summary : Support tools for GRUB. 2023-10-18T14:26:14.313 INFO:teuthology.orchestra.run.smithi158.stdout:Description : 2023-10-18T14:26:14.314 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-18T14:26:14.314 INFO:teuthology.orchestra.run.smithi158.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:26:14.314 INFO:teuthology.orchestra.run.smithi158.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:26:14.314 INFO:teuthology.orchestra.run.smithi158.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:26:14.314 INFO:teuthology.orchestra.run.smithi158.stdout:hardware devices. 2023-10-18T14:26:14.315 INFO:teuthology.orchestra.run.smithi158.stdout: 2023-10-18T14:26:14.315 INFO:teuthology.orchestra.run.smithi158.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:26:14.316 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:26:14.316 INFO:teuthology.task.kernel:Updating grub on smithi158 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:14.316 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:26:15.137 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2023-10-18T14:26:15.223 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2023-10-18T14:26:15.798 INFO:teuthology.orchestra.run.smithi158.stderr:Generating grub configuration file ... 2023-10-18T14:26:16.927 INFO:teuthology.orchestra.run.smithi172.stderr:done 2023-10-18T14:26:16.928 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-10-18T14:26:16.956 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.X6w5DFMHyL 2023-10-18T14:26:16.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.X6w5DFMHyL 2023-10-18T14:26:17.029 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.X6w5DFMHyL 2023-10-18T14:26:17.188 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.X6w5DFMHyL is 6KB 2023-10-18T14:26:17.237 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.X6w5DFMHyL 2023-10-18T14:26:17.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:26:17.324 INFO:teuthology.orchestra.run.smithi172.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T14:26:17.324 INFO:teuthology.orchestra.run.smithi172.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T14:26:17.324 INFO:teuthology.orchestra.run.smithi172.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T14:26:17.325 INFO:teuthology.orchestra.run.smithi172.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T14:26:17.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.439 INFO:teuthology.orchestra.run.smithi114.stderr:done 2023-10-18T14:26:17.441 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2023-10-18T14:26:17.455 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.HhrWNDQWof 2023-10-18T14:26:17.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HhrWNDQWof 2023-10-18T14:26:17.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2023-10-18T14:26:17.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.HhrWNDQWof 2023-10-18T14:26:17.527 INFO:teuthology.orchestra.run.smithi158.stderr:done 2023-10-18T14:26:17.530 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2023-10-18T14:26:17.544 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.OuOHzaW0QF 2023-10-18T14:26:17.545 DEBUG:teuthology.orchestra.run.smithi158:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OuOHzaW0QF 2023-10-18T14:26:17.637 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.OuOHzaW0QF 2023-10-18T14:26:17.680 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.HhrWNDQWof is 6KB 2023-10-18T14:26:17.728 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.HhrWNDQWof 2023-10-18T14:26:17.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:26:17.746 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.OuOHzaW0QF is 6KB 2023-10-18T14:26:17.795 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.OuOHzaW0QF 2023-10-18T14:26:17.809 DEBUG:teuthology.orchestra.run.smithi158:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:26:17.835 INFO:teuthology.orchestra.run.smithi114.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T14:26:17.836 INFO:teuthology.orchestra.run.smithi114.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T14:26:17.836 INFO:teuthology.orchestra.run.smithi114.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T14:26:17.837 INFO:teuthology.orchestra.run.smithi114.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T14:26:17.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.880 INFO:teuthology.orchestra.run.smithi158.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T14:26:17.881 INFO:teuthology.orchestra.run.smithi158.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T14:26:17.881 INFO:teuthology.orchestra.run.smithi158.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T14:26:17.881 INFO:teuthology.orchestra.run.smithi158.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T14:26:17.882 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T14:26:17.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2023-10-18T14:26:18.061 DEBUG:teuthology.orchestra.run.smithi158:> sudo shutdown -r now 2023-10-18T14:26:47.530 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-18T14:26:47.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-18T14:26:47.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:47.975 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-18T14:26:47.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-18T14:26:47.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:26:48.064 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-18T14:26:48.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-18T14:26:48.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:05.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-10-18T14:27:06.490 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-10-18T14:27:06.491 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2023-10-18T14:27:14.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-18T14:27:14.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:15.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-18T14:27:15.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:15.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-18T14:27:15.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:18.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-10-18T14:27:18.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-10-18T14:27:18.555 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2023-10-18T14:27:30.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-18T14:27:30.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:30.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-18T14:27:30.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:30.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-18T14:27:30.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:27:33.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-10-18T14:27:48.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-18T14:27:48.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:28:02.683 DEBUG:teuthology.orchestra.run.smithi114:> true 2023-10-18T14:28:04.115 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-18T14:28:04.116 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T14:28:04.116 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2023-10-18T14:28:04.149 INFO:teuthology.orchestra.run.smithi114.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:04.150 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:04.150 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:28:04.150 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T14:28:05.151 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-18T14:28:05.152 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:28:05.348 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2023-10-18T14:28:05.361 DEBUG:teuthology.parallel:result is None 2023-10-18T14:28:07.162 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2023-10-18T14:28:25.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-18T14:28:25.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:28:25.562 DEBUG:teuthology.orchestra.run.smithi158:> true 2023-10-18T14:28:25.903 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2023-10-18T14:28:25.903 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T14:28:25.903 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2023-10-18T14:28:25.917 INFO:teuthology.orchestra.run.smithi158.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:25.917 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:25.918 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:28:25.918 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T14:28:26.919 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-18T14:28:26.920 DEBUG:teuthology.orchestra.run.smithi158:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:28:27.063 INFO:teuthology.orchestra.run.smithi158.stdout:ttyS1 2023-10-18T14:28:27.072 DEBUG:teuthology.parallel:result is None 2023-10-18T14:28:30.062 DEBUG:teuthology.orchestra.remote:timed out 2023-10-18T14:28:45.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-18T14:28:45.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:28:45.463 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-10-18T14:28:45.699 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-18T14:28:45.699 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T14:28:45.700 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-10-18T14:28:45.715 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:45.715 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T14:28:45.715 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:28:45.716 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T14:28:46.717 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2023-10-18T14:28:46.717 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:28:46.804 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2023-10-18T14:28:46.818 DEBUG:teuthology.parallel:result is None 2023-10-18T14:28:46.819 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T14:28:46.827 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T14:28:46.827 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:28:46.830 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:28:46.833 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:28:46.851 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T14:28:46.858 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T14:28:46.866 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T14:28:46.866 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:28:46.889 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:28:46.891 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:28:46.925 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T14:28:46.932 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T14:28:46.932 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:28:46.960 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:28:46.963 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:28:46.990 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:46.998 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:47.002 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:47.003 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:47.010 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:47.014 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:28:47.015 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T14:28:47.022 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T14:28:47.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:28:47.047 DEBUG:teuthology.orchestra.run.smithi158:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:28:47.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:28:47.082 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T14:28:47.091 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T14:28:47.091 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:28:47.113 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:28:47.121 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:28:47.137 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:47.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:47.248 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:47.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:47.368 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-10-18T14:28:47.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:28:47.432 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:47.477 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:47.519 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:47.600 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:47.642 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2023-10-18T14:28:47.643 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:28:47.708 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:47.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:28:47.796 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:47.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:28:47.921 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-18T14:28:47.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:28:47.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2023-10-18T14:28:47.990 DEBUG:teuthology.orchestra.run.smithi158:> sudo service rsyslog restart 2023-10-18T14:28:47.992 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2023-10-18T14:28:48.044 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:28:48.048 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:28:48.072 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:28:48.524 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T14:28:48.532 INFO:teuthology.task.internal:Starting timer... 2023-10-18T14:28:48.532 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T14:28:48.546 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T14:28:48.554 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T14:28:48.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2023-10-18T14:28:48.615 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: [ OK ] 2023-10-18T14:28:48.616 DEBUG:teuthology.orchestra.run.smithi158:> sudo service auditd rotate 2023-10-18T14:28:48.674 INFO:teuthology.orchestra.run.smithi158.stdout:Rotating logs: [ OK ] 2023-10-18T14:28:48.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2023-10-18T14:28:48.742 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: [ OK ] 2023-10-18T14:28:48.744 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T14:28:48.745 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2023-10-18T14:28:48.764 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2023-10-18T14:28:48.765 DEBUG:teuthology.orchestra.run.smithi158:> /usr/sbin/getenforce 2023-10-18T14:28:48.788 INFO:teuthology.orchestra.run.smithi158.stdout:Permissive 2023-10-18T14:28:48.789 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2023-10-18T14:28:48.809 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2023-10-18T14:28:48.809 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi158.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2023-10-18T14:28:48.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:28:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:28:48.835 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:28:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:28:48.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:28:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:28:48.891 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T14:28:48.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:28:48.916 DEBUG:teuthology.orchestra.run.smithi158:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:28:48.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:28:48.967 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T14:28:48.980 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-18T14:28:48.986 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-18T14:28:49.124 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T14:28:49.138 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T14:28:49.140 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi158.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T14:38:56.303 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543/ansible_failures.yaml 2023-10-18T14:38:56.326 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Curl error (28): Timeout was reached for https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 [Operation timed out after 30001 milliseconds with 0 out of 0 bytes received]', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:38:56.402 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=414834700ca346fd9ed7984eb1993b76 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Curl error (28): Timeout was reached for https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 [Operation timed out after 30001 milliseconds with 0 out of 0 bytes received]', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:38:56.405 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T14:38:56.415 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T14:38:56.415 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T14:38:56.430 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:38:56.519 DEBUG:teuthology.orchestra.run.smithi158:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:38:56.607 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:38:56.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:38:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:56.727 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:38:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:56.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:38:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:56.789 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T14:38:56.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T14:38:56.809 INFO:teuthology.task.internal:Duration was 608.276611 seconds 2023-10-18T14:38:56.809 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T14:38:56.821 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T14:38:56.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:38:56.825 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:38:56.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:38:56.866 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:38:56.873 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:38:56.878 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:38:57.325 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T14:38:57.326 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2023-10-18T14:38:57.327 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:38:57.346 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi158.front.sepia.ceph.com 2023-10-18T14:38:57.346 DEBUG:teuthology.orchestra.run.smithi158:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:38:57.384 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2023-10-18T14:38:57.385 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:38:57.405 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T14:38:57.405 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:38:57.408 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:38:57.427 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:38:57.483 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T14:38:57.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:38:57.486 DEBUG:teuthology.orchestra.run.smithi158:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:38:57.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:38:57.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T14:38:57.596 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T14:38:57.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:38:57.621 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:38:57.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:38:57.672 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T14:38:57.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:57.685 DEBUG:teuthology.orchestra.run.smithi158:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:57.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:57.706 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2023-10-18T14:38:57.711 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = core 2023-10-18T14:38:57.736 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2023-10-18T14:38:57.750 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:57.773 DEBUG:teuthology.orchestra.run.smithi158:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:57.787 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:38:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:38:57.805 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T14:38:57.815 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T14:38:57.818 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543/remote/smithi114 2023-10-18T14:38:57.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:38:57.860 DEBUG:teuthology.misc:Transferring archived files from smithi158:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543/remote/smithi158 2023-10-18T14:38:57.861 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:38:57.901 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543/remote/smithi172 2023-10-18T14:38:57.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:38:57.949 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T14:38:57.949 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:38:57.952 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:38:57.955 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:38:57.998 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T14:38:58.006 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T14:38:58.007 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T14:38:58.016 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T14:38:58.017 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:38:58.019 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:38:58.021 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:38:58.031 INFO:teuthology.orchestra.run.smithi114.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:38 /home/ubuntu/cephtest 2023-10-18T14:38:58.033 INFO:teuthology.orchestra.run.smithi158.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:38 /home/ubuntu/cephtest 2023-10-18T14:38:58.051 INFO:teuthology.orchestra.run.smithi172.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:38 /home/ubuntu/cephtest 2023-10-18T14:38:58.052 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T14:38:58.063 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T14:38:58.087 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T14:38:58.112 DEBUG:teuthology.nuke:shortname: smithi114 2023-10-18T14:38:58.113 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:38:58.139 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:14.055551', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEyd4ZBBCFjtHi1KpvJEZ6aUvCCopLOk4ky3JtJ17F6ij9MbiUoePsyXLJ4i+2rGvHfAfA4PUF+8adBozY6Aeko='} 2023-10-18T14:38:58.144 DEBUG:teuthology.nuke:shortname: smithi158 2023-10-18T14:38:58.144 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:38:58.168 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:14.059010', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFrqcoDjUYn/LXidl8axWJyRs+XJ3/uOtDLNmfZRp34oFMKFpEAfTtLY6wiYC+w0YqEqucQATNt7n0tH4uHGYWc='} 2023-10-18T14:38:58.174 DEBUG:teuthology.nuke:shortname: smithi172 2023-10-18T14:38:58.174 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:38:58.195 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-18_13:55:44-smoke-quincy-release-distro-default-smithi/7431543', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:20:14.057425', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDtqD8ljRQetAY1Eb+0EyWFlHbkHF68fptbIqoJsT5LmtRbYpiBMe6owPxU6npT8RnN4tKmoBV5W2apfD19b8Yc='} 2023-10-18T14:38:58.208 INFO:teuthology.orchestra.console.smithi114:Power off 2023-10-18T14:38:58.209 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:38:58.231 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2023-10-18T14:38:58.232 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:38:58.246 INFO:teuthology.orchestra.console.smithi158:Power off 2023-10-18T14:38:58.247 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:38:58.260 INFO:teuthology.orchestra.console.smithi172:Power off 2023-10-18T14:38:58.261 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:38:58.271 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2023-10-18T14:38:58.272 DEBUG:teuthology.orchestra.console.smithi158:power off output: Chassis Power Control: Down/Off 2023-10-18T14:38:58.272 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:38:58.283 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2023-10-18T14:38:58.284 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:38:58.294 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2023-10-18T14:38:58.307 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-18T14:39:02.273 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:02.296 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:02.308 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:02.382 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2023-10-18T14:39:02.406 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2023-10-18T14:39:02.417 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-18T14:39:06.386 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:06.407 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:06.417 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:39:06.496 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2023-10-18T14:39:06.516 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2023-10-18T14:39:06.527 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2023-10-18T14:39:06.597 INFO:teuthology.orchestra.console.smithi114:Power off completed 2023-10-18T14:39:06.616 INFO:teuthology.orchestra.console.smithi158:Power off completed 2023-10-18T14:39:06.628 INFO:teuthology.orchestra.console.smithi172:Power off completed 2023-10-18T14:39:06.735 INFO:teuthology.run:Summary data: description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/kclient_workunit_suites_pjd}} duration: 608.276611328125 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Connection timed out'', ''Curl error (28): Timeout was reached for https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 [Operation timed out after 30001 milliseconds with 0 out of 0 bytes received]'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=414834700ca346fd9ed7984eb1993b76 status: dead success: false 2023-10-18T14:39:06.735 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:39:06.815 INFO:teuthology.run:DEAD