2023-10-18T20:26:35.508 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-18T20:26:35.509 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi --description fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} --verbose --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238 -- /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238/orig.config.yaml 2023-10-18T20:26:35.559 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T20:26:35.641 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238 branch: wip-vshankar-testing-pacific-20231016.091648 description: fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} email: vshankar@redhat.com first_in_suite: false job_id: '7429238' kernel: client: sha1: distro flavor: default kdb: true sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-vshankar-testing-pacific-20231016.091648 ceph: cephfs: fs: - name: a - name: b session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - overall HEALTH_ - \(MON_DOWN\) sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: cephfs_name: a client.1: cephfs_name: b install: ceph: flavor: default sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 kclient: client.0: cephfs_name: a client.1: cephfs_name: b selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-pacific-20231016.091648 sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 owner: scheduled_vshankar@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 7825 sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 sleep_before_teardown: 0 subset: 33/1024 suite: fs suite_branch: wip-vshankar-testing-pacific-20231016.091648 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMxFE+u9di/0EmfUF+W7MSxNPiWmZvWG+56TZKkzJ40eQ3mCpky/vyMHAXiiIiC8PSvH93tP/G1cSV1kqzucLyc= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFqDtqJ/JZ0g0GlRo+GtRLerk3j++upFUPa2ue09Ii9IoWOS55U6dGg36/LeS/yNutaG5KkK0DUi0nqQwQgruiM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - mon_thrash: check_mds_failover: true revive_delay: 20 thrash_delay: 10 - workunit: clients: all: - fs/snaps teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_12:48:28 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T20:26:35.642 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa; will attempt to use it 2023-10-18T20:26:35.642 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5/qa/tasks 2023-10-18T20:26:35.643 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T20:26:35.644 INFO:teuthology.task.internal:Checking packages... 2023-10-18T20:26:35.667 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5' 2023-10-18T20:26:35.667 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T20:26:35.668 INFO:teuthology.packaging:ref: None 2023-10-18T20:26:35.668 INFO:teuthology.packaging:tag: None 2023-10-18T20:26:35.668 INFO:teuthology.packaging:branch: wip-vshankar-testing-pacific-20231016.091648 2023-10-18T20:26:35.669 INFO:teuthology.packaging:sha1: b4ddd52ca4ac2ca6dc3b66adb82565f632d0f5a5 2023-10-18T20:26:35.669 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-pacific-20231016.091648 2023-10-18T20:26:35.899 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-100.gb4ddd52c 2023-10-18T20:26:35.899 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T20:26:35.905 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T20:26:35.906 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T20:26:35.911 INFO:teuthology.task.internal:Saving configuration 2023-10-18T20:26:35.929 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T20:26:35.934 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T20:26:35.959 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 20:21:31.194518', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMxFE+u9di/0EmfUF+W7MSxNPiWmZvWG+56TZKkzJ40eQ3mCpky/vyMHAXiiIiC8PSvH93tP/G1cSV1kqzucLyc='} 2023-10-18T20:26:35.978 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 20:21:31.196495', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFqDtqJ/JZ0g0GlRo+GtRLerk3j++upFUPa2ue09Ii9IoWOS55U6dGg36/LeS/yNutaG5KkK0DUi0nqQwQgruiM='} 2023-10-18T20:26:35.979 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T20:26:35.985 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-18T20:26:35.985 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2023-10-18T20:26:35.986 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T20:26:36.049 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4eee8799d0>, signals=[15]) 2023-10-18T20:26:36.050 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T20:26:36.057 INFO:teuthology.task.internal:Opening connections... 2023-10-18T20:26:36.057 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2023-10-18T20:26:36.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:26:36.124 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2023-10-18T20:26:36.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:26:36.190 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T20:26:36.197 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2023-10-18T20:26:36.214 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2023-10-18T20:26:36.215 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2023-10-18T20:26:36.270 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T20:26:36.270 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T20:26:36.271 INFO:teuthology.orchestra.run.smithi069.stdout:ID="rhel" 2023-10-18T20:26:36.271 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="fedora" 2023-10-18T20:26:36.271 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="8.4" 2023-10-18T20:26:36.271 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el8" 2023-10-18T20:26:36.271 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T20:26:36.272 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2023-10-18T20:26:36.272 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T20:26:36.272 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T20:26:36.272 INFO:teuthology.orchestra.run.smithi069.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T20:26:36.272 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T20:26:36.272 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-18T20:26:36.273 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T20:26:36.273 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T20:26:36.273 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T20:26:36.273 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T20:26:36.274 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2023-10-18T20:26:36.295 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2023-10-18T20:26:36.312 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2023-10-18T20:26:36.312 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2023-10-18T20:26:36.367 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T20:26:36.368 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="8.4 (Ootpa)" 2023-10-18T20:26:36.368 INFO:teuthology.orchestra.run.smithi177.stdout:ID="rhel" 2023-10-18T20:26:36.368 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="fedora" 2023-10-18T20:26:36.368 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="8.4" 2023-10-18T20:26:36.369 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el8" 2023-10-18T20:26:36.369 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-18T20:26:36.369 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2023-10-18T20:26:36.369 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-18T20:26:36.369 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T20:26:36.370 INFO:teuthology.orchestra.run.smithi177.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T20:26:36.370 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T20:26:36.370 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-18T20:26:36.370 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T20:26:36.370 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T20:26:36.371 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-18T20:26:36.371 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2023-10-18T20:26:36.395 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T20:26:36.405 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T20:26:36.417 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T20:26:36.418 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2023-10-18T20:26:36.421 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2023-10-18T20:26:36.437 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T20:26:36.443 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T20:26:36.443 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2023-10-18T20:26:36.477 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2023-10-18T20:26:36.490 INFO:teuthology.orchestra.run.smithi069.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T20:26:36.493 INFO:teuthology.orchestra.run.smithi177.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-18T20:26:36.494 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T20:26:36.549 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T20:26:36.564 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'} 2023-10-18T20:26:36.564 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}} 2023-10-18T20:26:36.565 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-10-18T20:26:36.565 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T20:26:36.566 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T20:26:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:26:36.581 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-10-18T20:26:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:26:36.582 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2023-10-18T20:26:36.637 INFO:teuthology.orchestra.run.smithi069.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T20:26:36.637 INFO:teuthology.task.kernel:Running kernel on smithi069: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T20:26:36.638 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2023-10-18T20:26:36.639 INFO:teuthology.orchestra.run.smithi177.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-18T20:26:36.639 INFO:teuthology.task.kernel:Running kernel on smithi177: 4.18.0-305.25.1.el8_4.x86_64 2023-10-18T20:26:36.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2023-10-18T20:26:38.074 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-10-18T20:26:38.074 INFO:teuthology.orchestra.run.smithi069.stdout:Unable to read consumer identity 2023-10-18T20:26:38.125 INFO:teuthology.orchestra.run.smithi177.stdout:Updating Subscription Management repositories. 2023-10-18T20:26:38.125 INFO:teuthology.orchestra.run.smithi177.stdout:Unable to read consumer identity 2023-10-18T20:26:39.495 INFO:teuthology.orchestra.run.smithi069.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2023-10-18T20:26:39.578 INFO:teuthology.orchestra.run.smithi177.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2023-10-18T20:26:40.732 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-10-18T20:26:41.598 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 16 MB 00:01 2023-10-18T20:26:44.016 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2023-10-18T20:26:44.794 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T20:26:44.794 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T20:26:44.794 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T20:26:44.906 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-10-18T20:26:44.946 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2023-10-18T20:26:44.947 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2023-10-18T20:26:44.948 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2023-10-18T20:26:45.043 DEBUG:teuthology.orchestra.run.smithi069:> echo no | sudo yum reinstall kernel || true 2023-10-18T20:26:45.387 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-10-18T20:26:45.387 INFO:teuthology.orchestra.run.smithi069.stdout:Unable to read consumer identity 2023-10-18T20:26:45.677 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:01 ago on Wed 18 Oct 2023 08:26:44 PM UTC. 2023-10-18T20:26:45.682 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-18T20:26:45.682 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-18T20:26:45.683 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-18T20:26:45.808 INFO:teuthology.orchestra.run.smithi069.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T20:26:45.808 INFO:teuthology.orchestra.run.smithi069.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:26:45.808 INFO:teuthology.orchestra.run.smithi069.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:26:45.829 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2023-10-18T20:26:45.830 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2023-10-18T20:26:45.831 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2023-10-18T20:26:45.840 INFO:teuthology.orchestra.run.smithi069.stderr:Error: No packages marked for reinstall. 2023-10-18T20:26:45.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum reinstall -y kernel || true 2023-10-18T20:26:45.952 DEBUG:teuthology.orchestra.run.smithi177:> echo no | sudo yum reinstall kernel || true 2023-10-18T20:26:46.233 INFO:teuthology.orchestra.run.smithi069.stdout:Updating Subscription Management repositories. 2023-10-18T20:26:46.233 INFO:teuthology.orchestra.run.smithi069.stdout:Unable to read consumer identity 2023-10-18T20:26:46.335 INFO:teuthology.orchestra.run.smithi177.stdout:Updating Subscription Management repositories. 2023-10-18T20:26:46.336 INFO:teuthology.orchestra.run.smithi177.stdout:Unable to read consumer identity 2023-10-18T20:26:46.519 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 08:26:44 PM UTC. 2023-10-18T20:26:46.626 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 08:26:44 PM UTC. 2023-10-18T20:26:46.767 INFO:teuthology.orchestra.run.smithi177.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T20:26:46.768 INFO:teuthology.orchestra.run.smithi177.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:26:46.768 INFO:teuthology.orchestra.run.smithi177.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:26:46.786 INFO:teuthology.orchestra.run.smithi177.stderr:Error: No packages marked for reinstall. 2023-10-18T20:26:46.797 INFO:teuthology.orchestra.run.smithi069.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T20:26:46.797 INFO:teuthology.orchestra.run.smithi069.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:26:46.797 INFO:teuthology.orchestra.run.smithi069.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:26:46.813 INFO:teuthology.orchestra.run.smithi069.stderr:Error: No packages marked for reinstall. 2023-10-18T20:26:46.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum reinstall -y kernel || true 2023-10-18T20:26:46.869 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T20:26:46.994 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:46.995 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:46.995 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:46.995 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T20:26:46.996 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T20:26:46.996 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-18T20:26:46.996 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T20:26:46.996 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T20:26:47.121 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:47.121 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:47.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2023-10-18T20:26:47.149 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2023-10-18T20:26:47.149 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2023-10-18T20:26:47.149 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.02 2023-10-18T20:26:47.150 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 99.el8_4.1 2023-10-18T20:26:47.150 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2023-10-18T20:26:47.150 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T20:26:47.150 INFO:teuthology.orchestra.run.smithi069.stdout:Group : System Environment/Base 2023-10-18T20:26:47.151 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 9264631 2023-10-18T20:26:47.151 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2023-10-18T20:26:47.151 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T20:26:47.151 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T20:26:47.151 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T20:26:47.152 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T20:26:47.152 INFO:teuthology.orchestra.run.smithi069.stdout:Relocations : (not relocatable) 2023-10-18T20:26:47.152 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : Red Hat, Inc. 2023-10-18T20:26:47.152 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : Red Hat, Inc. 2023-10-18T20:26:47.153 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T20:26:47.153 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2023-10-18T20:26:47.153 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2023-10-18T20:26:47.153 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-18T20:26:47.153 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T20:26:47.154 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T20:26:47.154 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T20:26:47.154 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2023-10-18T20:26:47.154 INFO:teuthology.orchestra.run.smithi069.stdout: 2023-10-18T20:26:47.155 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T20:26:47.155 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T20:26:47.156 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:47.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T20:26:47.178 INFO:teuthology.orchestra.run.smithi177.stdout:Updating Subscription Management repositories. 2023-10-18T20:26:47.179 INFO:teuthology.orchestra.run.smithi177.stdout:Unable to read consumer identity 2023-10-18T20:26:47.466 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:03 ago on Wed 18 Oct 2023 08:26:44 PM UTC. 2023-10-18T20:26:47.599 INFO:teuthology.orchestra.run.smithi177.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-18T20:26:47.600 INFO:teuthology.orchestra.run.smithi177.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:26:47.600 INFO:teuthology.orchestra.run.smithi177.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-18T20:26:47.613 INFO:teuthology.orchestra.run.smithi177.stderr:Error: No packages marked for reinstall. 2023-10-18T20:26:47.663 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T20:26:47.788 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:47.788 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:47.788 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:47.788 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T20:26:47.789 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T20:26:47.789 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2023-10-18T20:26:47.789 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T20:26:47.789 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T20:26:47.914 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:47.915 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:47.915 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2023-10-18T20:26:47.942 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2023-10-18T20:26:47.942 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2023-10-18T20:26:47.942 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.02 2023-10-18T20:26:47.942 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 99.el8_4.1 2023-10-18T20:26:47.942 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2023-10-18T20:26:47.943 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-18T20:26:47.943 INFO:teuthology.orchestra.run.smithi177.stdout:Group : System Environment/Base 2023-10-18T20:26:47.943 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 9264631 2023-10-18T20:26:47.943 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2023-10-18T20:26:47.943 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-18T20:26:47.943 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-18T20:26:47.944 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-18T20:26:47.944 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-18T20:26:47.944 INFO:teuthology.orchestra.run.smithi177.stdout:Relocations : (not relocatable) 2023-10-18T20:26:47.944 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : Red Hat, Inc. 2023-10-18T20:26:47.944 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : Red Hat, Inc. 2023-10-18T20:26:47.945 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T20:26:47.945 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2023-10-18T20:26:47.945 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2023-10-18T20:26:47.945 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-18T20:26:47.945 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T20:26:47.945 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T20:26:47.946 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T20:26:47.946 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2023-10-18T20:26:47.946 INFO:teuthology.orchestra.run.smithi177.stdout: 2023-10-18T20:26:47.946 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T20:26:47.947 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T20:26:47.947 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:47.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T20:26:48.343 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2023-10-18T20:26:49.372 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2023-10-18T20:26:50.404 INFO:teuthology.orchestra.run.smithi069.stderr:done 2023-10-18T20:26:50.405 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2023-10-18T20:26:50.430 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.slAC1vgObQ 2023-10-18T20:26:50.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.slAC1vgObQ 2023-10-18T20:26:50.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.slAC1vgObQ 2023-10-18T20:26:50.662 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.slAC1vgObQ is 6KB 2023-10-18T20:26:50.710 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.slAC1vgObQ 2023-10-18T20:26:50.723 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T20:26:50.784 INFO:teuthology.orchestra.run.smithi069.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T20:26:50.785 INFO:teuthology.orchestra.run.smithi069.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T20:26:50.785 INFO:teuthology.orchestra.run.smithi069.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T20:26:50.785 INFO:teuthology.orchestra.run.smithi069.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T20:26:50.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:50.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2023-10-18T20:26:51.529 INFO:teuthology.orchestra.run.smithi177.stderr:done 2023-10-18T20:26:51.530 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2023-10-18T20:26:51.544 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.YGZmpnDM2s 2023-10-18T20:26:51.545 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YGZmpnDM2s 2023-10-18T20:26:51.627 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.YGZmpnDM2s 2023-10-18T20:26:51.794 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.YGZmpnDM2s is 6KB 2023-10-18T20:26:51.842 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.YGZmpnDM2s 2023-10-18T20:26:51.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T20:26:51.923 INFO:teuthology.orchestra.run.smithi177.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-18T20:26:51.924 INFO:teuthology.orchestra.run.smithi177.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-18T20:26:51.924 INFO:teuthology.orchestra.run.smithi177.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-18T20:26:51.924 INFO:teuthology.orchestra.run.smithi177.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-18T20:26:51.925 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-18T20:26:52.103 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2023-10-18T20:27:20.986 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-18T20:27:20.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-18T20:27:20.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:27:22.107 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-10-18T20:27:22.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-10-18T20:27:22.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:28:21.047 DEBUG:teuthology.orchestra.remote:timed out 2023-10-18T20:28:22.110 DEBUG:teuthology.orchestra.remote:timed out 2023-10-18T20:28:30.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-18T20:28:30.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:28:30.417 DEBUG:teuthology.orchestra.run.smithi069:> true 2023-10-18T20:28:30.645 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2023-10-18T20:28:30.645 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T20:28:30.646 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2023-10-18T20:28:30.701 INFO:teuthology.orchestra.run.smithi069.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T20:28:30.701 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:28:30.701 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T20:28:30.702 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T20:28:31.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-10-18T20:28:31.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T20:28:31.486 DEBUG:teuthology.orchestra.run.smithi177:> true 2023-10-18T20:28:31.703 DEBUG:teuthology.parallel:result is None 2023-10-18T20:28:31.807 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-10-18T20:28:31.807 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-18T20:28:31.808 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2023-10-18T20:28:31.823 INFO:teuthology.orchestra.run.smithi177.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-18T20:28:31.823 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-18T20:28:31.823 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T20:28:31.824 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-18T20:28:32.825 DEBUG:teuthology.parallel:result is None 2023-10-18T20:28:32.825 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T20:28:32.837 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T20:28:32.838 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T20:28:32.843 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T20:28:32.858 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T20:28:32.870 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T20:28:32.884 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T20:28:32.885 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T20:28:32.900 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T20:28:32.924 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T20:28:32.936 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T20:28:32.936 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T20:28:32.961 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T20:28:33.136 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T20:28:33.162 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T20:28:33.164 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T20:28:33.176 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T20:28:33.178 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T20:28:33.189 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T20:28:33.189 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T20:28:33.192 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T20:28:33.217 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T20:28:33.226 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T20:28:33.227 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T20:28:33.256 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T20:28:33.271 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T20:28:33.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T20:28:33.397 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T20:28:33.480 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T20:28:33.518 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2023-10-18T20:28:33.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T20:28:33.606 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T20:28:33.648 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T20:28:33.714 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T20:28:33.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T20:28:33.796 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2023-10-18T20:28:33.796 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T20:28:33.869 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2023-10-18T20:28:33.872 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2023-10-18T20:28:33.928 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T20:28:33.973 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T20:28:34.394 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T20:28:34.403 INFO:teuthology.task.internal:Starting timer... 2023-10-18T20:28:34.403 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T20:28:34.416 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T20:28:34.425 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T20:28:34.426 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2023-10-18T20:28:34.539 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: [ OK ] 2023-10-18T20:28:34.541 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2023-10-18T20:28:34.644 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: [ OK ] 2023-10-18T20:28:34.646 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T20:28:34.646 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2023-10-18T20:28:34.677 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2023-10-18T20:28:34.678 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2023-10-18T20:28:34.701 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2023-10-18T20:28:34.702 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2023-10-18T20:28:34.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T20:28:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:28:34.746 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T20:28:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:28:34.774 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T20:28:34.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2023-10-18T20:28:34.824 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2023-10-18T20:28:34.850 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T20:28:34.861 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-18T20:28:34.862 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T20:28:34.879 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T20:28:34.880 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T20:37:19.610 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238/ansible_failures.yaml 2023-10-18T20:37:19.612 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T20:37:19.685 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=314d4de26c2943e28615d445b0e9793f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T20:37:19.687 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T20:37:19.696 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T20:37:19.696 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T20:37:19.705 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T20:37:19.803 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T20:37:19.899 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T20:37:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:37:19.928 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T20:37:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:37:19.958 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T20:37:19.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T20:37:19.974 INFO:teuthology.task.internal:Duration was 525.570571 seconds 2023-10-18T20:37:19.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T20:37:19.987 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T20:37:19.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T20:37:19.990 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T20:37:20.032 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T20:37:20.041 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T20:37:20.396 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T20:37:20.396 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2023-10-18T20:37:20.397 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T20:37:20.455 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2023-10-18T20:37:20.456 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T20:37:20.474 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T20:37:20.475 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T20:37:20.498 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T20:37:20.550 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T20:37:20.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T20:37:20.574 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T20:37:20.656 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T20:37:20.665 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T20:37:20.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T20:37:20.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T20:37:20.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T20:37:20.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T20:37:20.733 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T20:37:20.754 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2023-10-18T20:37:20.778 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2023-10-18T20:37:20.792 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T20:37:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:37:20.822 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T20:37:20.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T20:37:20.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T20:37:20.854 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T20:37:20.856 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238/remote/smithi069 2023-10-18T20:37:20.857 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T20:37:20.916 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238/remote/smithi177 2023-10-18T20:37:20.917 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T20:37:20.960 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T20:37:20.960 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T20:37:20.963 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T20:37:21.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T20:37:21.020 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T20:37:21.020 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T20:37:21.029 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T20:37:21.030 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T20:37:21.032 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T20:37:21.044 INFO:teuthology.orchestra.run.smithi069.stdout: 262383 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 20:37 /home/ubuntu/cephtest 2023-10-18T20:37:21.061 INFO:teuthology.orchestra.run.smithi177.stdout: 262396 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 20:37 /home/ubuntu/cephtest 2023-10-18T20:37:21.063 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T20:37:21.072 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T20:37:21.091 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T20:37:21.115 DEBUG:teuthology.nuke:shortname: smithi069 2023-10-18T20:37:21.115 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T20:37:21.136 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 20:21:31.194518', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMxFE+u9di/0EmfUF+W7MSxNPiWmZvWG+56TZKkzJ40eQ3mCpky/vyMHAXiiIiC8PSvH93tP/G1cSV1kqzucLyc='} 2023-10-18T20:37:21.140 DEBUG:teuthology.nuke:shortname: smithi177 2023-10-18T20:37:21.141 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T20:37:21.162 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-10-16_12:48:28-fs-wip-vshankar-testing-pacific-20231016.091648-testing-default-smithi/7429238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 20:21:31.196495', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFqDtqJ/JZ0g0GlRo+GtRLerk3j++upFUPa2ue09Ii9IoWOS55U6dGg36/LeS/yNutaG5KkK0DUi0nqQwQgruiM='} 2023-10-18T20:37:21.188 INFO:teuthology.orchestra.console.smithi069:Power off 2023-10-18T20:37:21.189 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T20:37:21.215 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2023-10-18T20:37:21.216 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:37:21.231 INFO:teuthology.orchestra.console.smithi177:Power off 2023-10-18T20:37:21.232 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T20:37:21.241 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-10-18T20:37:21.254 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2023-10-18T20:37:21.254 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:37:21.277 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2023-10-18T20:37:25.244 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:37:25.279 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:37:25.356 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-10-18T20:37:25.390 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2023-10-18T20:37:29.360 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:37:29.391 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:37:29.471 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2023-10-18T20:37:29.502 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2023-10-18T20:37:33.474 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:37:33.504 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T20:37:33.585 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2023-10-18T20:37:33.615 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2023-10-18T20:37:33.686 INFO:teuthology.orchestra.console.smithi069:Power off completed 2023-10-18T20:37:33.716 INFO:teuthology.orchestra.console.smithi177:Power off completed 2023-10-18T20:37:33.826 INFO:teuthology.run:Summary data: description: fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} duration: 525.5705709457397 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Connection timed out'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=314d4de26c2943e28615d445b0e9793f status: dead success: false 2023-10-18T20:37:33.826 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T20:37:33.920 INFO:teuthology.run:DEAD