2023-10-18T20:24:28.771 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-18T20:24:28.772 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_vshankar@teuthology --verbose --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} --archive /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237 --name vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi -- /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237/orig.config.yaml 2023-10-18T20:24:28.810 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T20:24:28.890 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237 branch: wip-vshankar-testing-pacific-20231016.091648 description: fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} email: vshankar@redhat.com first_in_suite: false job_id: '7429237' kernel: client: sha1: distro flavor: default kdb: true sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-vshankar-testing-pacific-20231016.091648 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-pacific-20231016.091648 sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 owner: scheduled_vshankar@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7825 sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 sleep_before_teardown: 0 subset: 33/1024 suite: fs suite_branch: wip-vshankar-testing-pacific-20231016.091648 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 targets: smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcwCD2y7Yddqp0Uc2TT1t0At9YEbL1X28pfY6ff29v3+mLtBqNcFGWOvXWqc7Pf0DJ6xC3FRvBVNhmlwHzCa+g= smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxxC596dhoS/9vVRlYxXongm44nV3iJIlVSX9nYcn837aVZU0UTO3HgB7+tRX9taDLtMClmVvu9mt92TgPlX4I= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_journal_migration teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_12:48:28 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T20:24:28.890 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa; will attempt to use it 2023-10-18T20:24:28.891 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks 2023-10-18T20:24:28.891 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T20:24:28.893 INFO:teuthology.task.internal:Checking packages... 2023-10-18T20:24:28.932 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5' 2023-10-18T20:24:28.932 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T20:24:28.933 INFO:teuthology.packaging:ref: None 2023-10-18T20:24:28.933 INFO:teuthology.packaging:tag: None 2023-10-18T20:24:28.933 INFO:teuthology.packaging:branch: wip-vshankar-testing-pacific-20231016.091648 2023-10-18T20:24:28.934 INFO:teuthology.packaging:sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 2023-10-18T20:24:28.934 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-pacific-20231016.091648 2023-10-18T20:24:29.072 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-100.gb4ddd52c 2023-10-18T20:24:29.072 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T20:24:29.078 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T20:24:29.079 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T20:24:29.084 INFO:teuthology.task.internal:Saving configuration 2023-10-18T20:24:29.098 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T20:24:29.104 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T20:24:29.129 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 20:18:50.494837', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcwCD2y7Yddqp0Uc2TT1t0At9YEbL1X28pfY6ff29v3+mLtBqNcFGWOvXWqc7Pf0DJ6xC3FRvBVNhmlwHzCa+g='} 2023-10-18T20:24:29.146 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 20:18:50.496535', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxxC596dhoS/9vVRlYxXongm44nV3iJIlVSX9nYcn837aVZU0UTO3HgB7+tRX9taDLtMClmVvu9mt92TgPlX4I='} 2023-10-18T20:24:29.147 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T20:24:29.153 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-18T20:24:29.153 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-18T20:24:29.153 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T20:24:29.220 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8e93cb6a60>, signals=[15]) 2023-10-18T20:24:29.221 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T20:24:29.227 INFO:teuthology.task.internal:Opening connections... 2023-10-18T20:24:29.227 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2023-10-18T20:24:29.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:24:29.294 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2023-10-18T20:24:29.294 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:24:29.360 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T20:24:29.367 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2023-10-18T20:24:29.384 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2023-10-18T20:24:29.385 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2023-10-18T20:24:29.441 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T20:24:29.441 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T20:24:29.441 INFO:teuthology.orchestra.run.smithi118.stdout:ID="rhel" 2023-10-18T20:24:29.441 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="fedora" 2023-10-18T20:24:29.442 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8.4" 2023-10-18T20:24:29.442 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8" 2023-10-18T20:24:29.442 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T20:24:29.442 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2023-10-18T20:24:29.442 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T20:24:29.443 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T20:24:29.443 INFO:teuthology.orchestra.run.smithi118.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T20:24:29.443 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T20:24:29.443 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-18T20:24:29.443 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T20:24:29.443 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T20:24:29.444 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T20:24:29.444 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T20:24:29.444 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2023-10-18T20:24:29.472 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2023-10-18T20:24:29.489 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2023-10-18T20:24:29.489 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2023-10-18T20:24:29.546 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T20:24:29.546 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T20:24:29.546 INFO:teuthology.orchestra.run.smithi148.stdout:ID="rhel" 2023-10-18T20:24:29.546 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="fedora" 2023-10-18T20:24:29.546 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8.4" 2023-10-18T20:24:29.547 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2023-10-18T20:24:29.547 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T20:24:29.547 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2023-10-18T20:24:29.547 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T20:24:29.547 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T20:24:29.548 INFO:teuthology.orchestra.run.smithi148.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T20:24:29.548 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T20:24:29.548 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-18T20:24:29.548 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T20:24:29.549 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T20:24:29.549 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T20:24:29.549 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T20:24:29.549 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2023-10-18T20:24:29.572 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T20:24:29.581 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T20:24:29.586 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T20:24:29.586 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2023-10-18T20:24:29.589 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2023-10-18T20:24:29.605 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T20:24:29.611 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T20:24:29.611 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2023-10-18T20:24:29.645 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2023-10-18T20:24:29.658 INFO:teuthology.orchestra.run.smithi118.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T20:24:29.661 INFO:teuthology.orchestra.run.smithi148.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T20:24:29.661 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T20:24:29.714 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T20:24:29.727 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'} 2023-10-18T20:24:29.728 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-10-18T20:24:29.728 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-10-18T20:24:29.729 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T20:24:29.729 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T20:24:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:24:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:24:29.746 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-10-18T20:24:29.746 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2023-10-18T20:24:29.801 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T20:24:29.801 INFO:teuthology.task.kernel:Running kernel on smithi148: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T20:24:29.802 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2023-10-18T20:24:29.803 INFO:teuthology.orchestra.run.smithi118.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T20:24:29.803 INFO:teuthology.task.kernel:Running kernel on smithi118: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T20:24:29.803 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2023-10-18T20:24:31.202 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2023-10-18T20:24:31.203 INFO:teuthology.orchestra.run.smithi118.stdout:Unable to read consumer identity 2023-10-18T20:24:31.225 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2023-10-18T20:24:31.225 INFO:teuthology.orchestra.run.smithi148.stdout:Unable to read consumer identity 2023-10-18T20:24:32.615 INFO:teuthology.orchestra.run.smithi148.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-10-18T20:24:33.416 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2023-10-18T20:24:33.713 INFO:teuthology.orchestra.run.smithi118.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-10-18T20:24:34.370 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2023-10-18T20:24:36.738 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-10-18T20:24:37.474 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T20:24:37.474 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T20:24:37.474 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T20:24:37.621 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2023-10-18T20:24:37.622 INFO:teuthology.orchestra.run.smithi148.stdout:Nothing to do. 2023-10-18T20:24:37.622 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2023-10-18T20:24:37.710 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2023-10-18T20:24:37.729 DEBUG:teuthology.orchestra.run.smithi148:> echo no | sudo yum reinstall kernel || true 2023-10-18T20:24:38.073 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2023-10-18T20:24:38.074 INFO:teuthology.orchestra.run.smithi148.stdout:Unable to read consumer identity 2023-10-18T20:24:38.371 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 08:24:36 PM UTC. 2023-10-18T20:24:38.438 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T20:24:38.438 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T20:24:38.438 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T20:24:38.506 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T20:24:38.507 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:24:38.507 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:24:38.530 INFO:teuthology.orchestra.run.smithi148.stderr:Error: No packages marked for reinstall. 2023-10-18T20:24:38.581 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum reinstall -y kernel || true 2023-10-18T20:24:38.596 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2023-10-18T20:24:38.598 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2023-10-18T20:24:38.598 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2023-10-18T20:24:38.702 DEBUG:teuthology.orchestra.run.smithi118:> echo no | sudo yum reinstall kernel || true 2023-10-18T20:24:38.926 INFO:teuthology.orchestra.run.smithi148.stdout:Updating Subscription Management repositories. 2023-10-18T20:24:38.927 INFO:teuthology.orchestra.run.smithi148.stdout:Unable to read consumer identity 2023-10-18T20:24:39.049 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2023-10-18T20:24:39.049 INFO:teuthology.orchestra.run.smithi118.stdout:Unable to read consumer identity 2023-10-18T20:24:39.207 INFO:teuthology.orchestra.run.smithi148.stdout:Last metadata expiration check: 0:00:03 ago on Wed 18 Oct 2023 08:24:36 PM UTC. 2023-10-18T20:24:39.336 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 08:24:37 PM UTC. 2023-10-18T20:24:39.338 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T20:24:39.339 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:24:39.339 INFO:teuthology.orchestra.run.smithi148.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:24:39.355 INFO:teuthology.orchestra.run.smithi148.stderr:Error: No packages marked for reinstall. 2023-10-18T20:24:39.411 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T20:24:39.466 INFO:teuthology.orchestra.run.smithi118.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T20:24:39.467 INFO:teuthology.orchestra.run.smithi118.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:24:39.467 INFO:teuthology.orchestra.run.smithi118.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:24:39.499 INFO:teuthology.orchestra.run.smithi118.stderr:Error: No packages marked for reinstall. 2023-10-18T20:24:39.535 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:39.535 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:39.535 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:39.536 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T20:24:39.536 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T20:24:39.536 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2023-10-18T20:24:39.536 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T20:24:39.536 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T20:24:39.548 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum reinstall -y kernel || true 2023-10-18T20:24:39.660 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:39.661 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:39.661 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2023-10-18T20:24:39.687 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2023-10-18T20:24:39.688 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2023-10-18T20:24:39.688 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.02 2023-10-18T20:24:39.688 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 99.el8_4.1 2023-10-18T20:24:39.688 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2023-10-18T20:24:39.688 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T20:24:39.689 INFO:teuthology.orchestra.run.smithi148.stdout:Group : System Environment/Base 2023-10-18T20:24:39.689 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 9264631 2023-10-18T20:24:39.689 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2023-10-18T20:24:39.689 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T20:24:39.689 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T20:24:39.690 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T20:24:39.690 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T20:24:39.690 INFO:teuthology.orchestra.run.smithi148.stdout:Relocations : (not relocatable) 2023-10-18T20:24:39.690 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : Red Hat, Inc. 2023-10-18T20:24:39.690 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : Red Hat, Inc. 2023-10-18T20:24:39.691 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T20:24:39.691 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2023-10-18T20:24:39.691 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2023-10-18T20:24:39.691 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-18T20:24:39.691 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T20:24:39.692 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T20:24:39.692 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T20:24:39.692 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2023-10-18T20:24:39.692 INFO:teuthology.orchestra.run.smithi148.stdout: 2023-10-18T20:24:39.692 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T20:24:39.693 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T20:24:39.693 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:39.694 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T20:24:39.890 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2023-10-18T20:24:39.890 INFO:teuthology.orchestra.run.smithi118.stdout:Unable to read consumer identity 2023-10-18T20:24:40.180 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:03 ago on Wed 18 Oct 2023 08:24:37 PM UTC. 2023-10-18T20:24:40.307 INFO:teuthology.orchestra.run.smithi118.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T20:24:40.308 INFO:teuthology.orchestra.run.smithi118.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:24:40.308 INFO:teuthology.orchestra.run.smithi118.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:24:40.322 INFO:teuthology.orchestra.run.smithi118.stderr:Error: No packages marked for reinstall. 2023-10-18T20:24:40.377 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T20:24:40.501 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:40.502 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:40.502 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:40.502 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T20:24:40.503 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T20:24:40.503 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-18T20:24:40.503 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T20:24:40.503 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T20:24:40.628 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:40.629 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:40.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2023-10-18T20:24:40.656 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2023-10-18T20:24:40.656 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2023-10-18T20:24:40.656 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.02 2023-10-18T20:24:40.657 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 99.el8_4.1 2023-10-18T20:24:40.657 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2023-10-18T20:24:40.657 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T20:24:40.657 INFO:teuthology.orchestra.run.smithi118.stdout:Group : System Environment/Base 2023-10-18T20:24:40.657 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 9264631 2023-10-18T20:24:40.657 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2023-10-18T20:24:40.658 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T20:24:40.658 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T20:24:40.658 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T20:24:40.658 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T20:24:40.658 INFO:teuthology.orchestra.run.smithi118.stdout:Relocations : (not relocatable) 2023-10-18T20:24:40.659 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : Red Hat, Inc. 2023-10-18T20:24:40.659 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : Red Hat, Inc. 2023-10-18T20:24:40.659 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T20:24:40.659 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2023-10-18T20:24:40.659 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2023-10-18T20:24:40.660 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-18T20:24:40.660 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T20:24:40.660 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T20:24:40.660 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T20:24:40.660 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2023-10-18T20:24:40.661 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-18T20:24:40.661 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T20:24:40.662 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T20:24:40.662 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:40.662 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T20:24:40.974 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2023-10-18T20:24:42.174 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2023-10-18T20:24:43.339 INFO:teuthology.orchestra.run.smithi148.stderr:done 2023-10-18T20:24:43.340 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2023-10-18T20:24:43.354 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.JUKx2GEmtY 2023-10-18T20:24:43.355 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JUKx2GEmtY 2023-10-18T20:24:43.425 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.JUKx2GEmtY 2023-10-18T20:24:43.591 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.JUKx2GEmtY is 6KB 2023-10-18T20:24:43.639 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.JUKx2GEmtY 2023-10-18T20:24:43.652 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T20:24:43.738 INFO:teuthology.orchestra.run.smithi148.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T20:24:43.738 INFO:teuthology.orchestra.run.smithi148.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T20:24:43.738 INFO:teuthology.orchestra.run.smithi148.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T20:24:43.738 INFO:teuthology.orchestra.run.smithi148.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T20:24:43.739 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:43.861 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2023-10-18T20:24:44.286 INFO:teuthology.orchestra.run.smithi118.stderr:done 2023-10-18T20:24:44.287 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2023-10-18T20:24:44.301 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.vgov0Hlgjw 2023-10-18T20:24:44.302 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vgov0Hlgjw 2023-10-18T20:24:44.374 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.vgov0Hlgjw 2023-10-18T20:24:44.540 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.vgov0Hlgjw is 6KB 2023-10-18T20:24:44.587 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.vgov0Hlgjw 2023-10-18T20:24:44.601 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T20:24:44.672 INFO:teuthology.orchestra.run.smithi118.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T20:24:44.672 INFO:teuthology.orchestra.run.smithi118.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T20:24:44.673 INFO:teuthology.orchestra.run.smithi118.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T20:24:44.673 INFO:teuthology.orchestra.run.smithi118.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T20:24:44.674 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:24:44.843 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2023-10-18T20:25:13.890 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-10-18T20:25:13.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-18T20:25:13.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:25:14.847 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-18T20:25:14.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-18T20:25:14.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:25:48.410 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-10-18T20:25:49.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2023-10-18T20:25:57.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-18T20:25:57.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:25:58.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-18T20:25:58.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:26:15.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-10-18T20:26:27.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-18T20:26:27.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:26:29.660 DEBUG:teuthology.orchestra.run.smithi148:> true 2023-10-18T20:26:29.924 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2023-10-18T20:26:29.925 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T20:26:29.926 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2023-10-18T20:26:29.981 INFO:teuthology.orchestra.run.smithi148.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:29.981 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:29.982 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T20:26:29.982 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T20:26:30.984 DEBUG:teuthology.parallel:result is None 2023-10-18T20:26:58.440 DEBUG:teuthology.orchestra.remote:timed out 2023-10-18T20:27:10.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-18T20:27:10.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:27:10.740 DEBUG:teuthology.orchestra.run.smithi118:> true 2023-10-18T20:27:10.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-18T20:27:10.969 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T20:27:10.970 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2023-10-18T20:27:10.984 INFO:teuthology.orchestra.run.smithi118.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T20:27:10.984 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:27:10.985 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T20:27:10.985 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T20:27:11.986 DEBUG:teuthology.parallel:result is None 2023-10-18T20:27:11.987 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T20:27:11.995 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T20:27:11.995 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T20:27:11.997 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T20:27:12.019 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T20:27:12.026 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T20:27:12.032 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T20:27:12.032 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T20:27:12.052 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T20:27:12.093 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T20:27:12.099 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T20:27:12.100 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T20:27:12.117 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T20:27:12.240 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T20:27:12.254 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T20:27:12.260 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T20:27:12.272 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T20:27:12.274 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T20:27:12.282 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T20:27:12.282 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T20:27:12.285 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T20:27:12.312 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T20:27:12.324 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T20:27:12.325 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T20:27:12.352 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T20:27:12.367 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T20:27:12.449 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T20:27:12.511 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T20:27:12.550 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T20:27:12.591 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-18T20:27:12.591 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T20:27:12.655 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T20:27:12.698 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T20:27:12.765 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T20:27:12.806 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T20:27:12.847 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2023-10-18T20:27:12.847 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T20:27:12.911 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2023-10-18T20:27:12.913 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2023-10-18T20:27:12.959 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T20:27:13.014 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T20:27:13.216 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T20:27:13.224 INFO:teuthology.task.internal:Starting timer... 2023-10-18T20:27:13.224 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T20:27:13.233 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T20:27:13.241 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T20:27:13.242 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2023-10-18T20:27:13.356 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: [ OK ] 2023-10-18T20:27:13.358 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2023-10-18T20:27:13.413 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: [ OK ] 2023-10-18T20:27:13.414 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T20:27:13.415 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2023-10-18T20:27:13.443 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2023-10-18T20:27:13.445 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2023-10-18T20:27:13.463 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2023-10-18T20:27:13.464 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi118.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive'} 2023-10-18T20:27:13.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T20:27:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:27:13.508 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T20:27:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:27:13.534 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T20:27:13.535 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2023-10-18T20:27:13.573 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2023-10-18T20:27:13.600 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T20:27:13.611 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-18T20:27:13.611 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T20:27:13.625 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T20:27:13.627 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi118.front.sepia.ceph.com,smithi148.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T20:36:03.217 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237/ansible_failures.yaml 2023-10-18T20:36:03.220 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T20:36:03.295 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ab3362bbbcb14d0c840a2eb8f262fcd7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T20:36:03.297 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T20:36:03.308 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T20:36:03.308 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T20:36:03.323 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T20:36:03.429 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T20:36:03.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T20:36:03.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:36:03.551 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T20:36:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:36:03.581 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T20:36:03.590 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T20:36:03.604 INFO:teuthology.task.internal:Duration was 530.379700 seconds 2023-10-18T20:36:03.604 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T20:36:03.618 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T20:36:03.618 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T20:36:03.621 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T20:36:03.662 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T20:36:03.665 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T20:36:03.966 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T20:36:03.967 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2023-10-18T20:36:03.967 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T20:36:04.025 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2023-10-18T20:36:04.026 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T20:36:04.045 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T20:36:04.046 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T20:36:04.069 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T20:36:04.120 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T20:36:04.121 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T20:36:04.142 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T20:36:04.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T20:36:04.234 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T20:36:04.235 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T20:36:04.260 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T20:36:04.284 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T20:36:04.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T20:36:04.302 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T20:36:04.323 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2023-10-18T20:36:04.348 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2023-10-18T20:36:04.361 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T20:36:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:36:04.391 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T20:36:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:36:04.416 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T20:36:04.427 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T20:36:04.429 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237/remote/smithi118 2023-10-18T20:36:04.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T20:36:04.484 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237/remote/smithi148 2023-10-18T20:36:04.486 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T20:36:04.538 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T20:36:04.539 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T20:36:04.541 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T20:36:04.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T20:36:04.597 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T20:36:04.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T20:36:04.611 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T20:36:04.612 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T20:36:04.615 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T20:36:04.627 INFO:teuthology.orchestra.run.smithi118.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 20:36 /home/ubuntu/cephtest 2023-10-18T20:36:04.641 INFO:teuthology.orchestra.run.smithi148.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 20:36 /home/ubuntu/cephtest 2023-10-18T20:36:04.643 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T20:36:04.652 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T20:36:04.673 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T20:36:04.700 DEBUG:teuthology.nuke:shortname: smithi118 2023-10-18T20:36:04.700 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T20:36:04.724 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 20:18:50.494837', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBcwCD2y7Yddqp0Uc2TT1t0At9YEbL1X28pfY6ff29v3+mLtBqNcFGWOvXWqc7Pf0DJ6xC3FRvBVNhmlwHzCa+g='} 2023-10-18T20:36:04.728 DEBUG:teuthology.nuke:shortname: smithi148 2023-10-18T20:36:04.728 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T20:36:04.750 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429237', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 20:18:50.496535', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxxC596dhoS/9vVRlYxXongm44nV3iJIlVSX9nYcn837aVZU0UTO3HgB7+tRX9taDLtMClmVvu9mt92TgPlX4I='} 2023-10-18T20:36:04.777 INFO:teuthology.orchestra.console.smithi118:Power off 2023-10-18T20:36:04.778 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T20:36:04.804 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2023-10-18T20:36:04.804 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:36:04.820 INFO:teuthology.orchestra.console.smithi148:Power off 2023-10-18T20:36:04.821 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T20:36:04.830 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-18T20:36:04.843 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2023-10-18T20:36:04.844 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:36:04.869 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-10-18T20:36:08.833 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:36:08.871 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:36:08.943 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-18T20:36:08.981 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-10-18T20:36:12.946 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:36:12.983 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:36:13.056 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2023-10-18T20:36:13.093 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-10-18T20:36:13.158 INFO:teuthology.orchestra.console.smithi118:Power off completed 2023-10-18T20:36:13.195 INFO:teuthology.orchestra.console.smithi148:Power off completed 2023-10-18T20:36:13.302 INFO:teuthology.run:Summary data: description: fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} duration: 530.3797001838684 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Connection timed out'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ab3362bbbcb14d0c840a2eb8f262fcd7 status: dead success: false 2023-10-18T20:36:13.303 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T20:36:13.389 INFO:teuthology.run:DEAD