2023-10-18T14:34:32.299 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-18T14:34:32.300 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi --verbose --description fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} --archive /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909 --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909/orig.config.yaml 2023-10-18T14:34:32.354 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:34:32.454 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909 branch: wip-batrick-testing-20231016.203825 description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} email: pdonnell@redhat.com first_in_suite: false job_id: '7430909' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-batrick-testing-20231016.203825 ceph: cephfs: fs: - name: a - name: b session_timeout: 300 cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: client: client_shutdown_timeout: 120 global: ms die on skipped message: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - overall HEALTH_ - \(MON_DOWN\) sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: cephfs_name: a client.1: cephfs_name: b install: ceph: flavor: default sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb kclient: client.0: cephfs_name: a client.1: cephfs_name: b syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20231016.203825 sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 7703 sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb sleep_before_teardown: 0 subset: 153/256 suite: fs suite_branch: wip-batrick-testing-20231016.203825 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - mon_thrash: check_mds_failover: true revive_delay: 20 thrash_delay: 10 - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_23:59:58 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T14:34:32.455 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa; will attempt to use it 2023-10-18T14:34:32.456 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa/tasks 2023-10-18T14:34:32.456 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T14:34:32.458 INFO:teuthology.task.internal:Checking packages... 2023-10-18T14:34:32.499 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'f7420d7f8e55ceea786dcbf536c338ce920c21eb' 2023-10-18T14:34:32.499 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T14:34:32.500 INFO:teuthology.packaging:ref: None 2023-10-18T14:34:32.500 INFO:teuthology.packaging:tag: None 2023-10-18T14:34:32.501 INFO:teuthology.packaging:branch: wip-batrick-testing-20231016.203825 2023-10-18T14:34:32.501 INFO:teuthology.packaging:sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb 2023-10-18T14:34:32.501 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-batrick-testing-20231016.203825 2023-10-18T14:34:32.652 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6786.gf7420d7f 2023-10-18T14:34:32.652 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T14:34:32.658 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T14:34:32.658 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T14:34:32.708 INFO:teuthology.task.internal:Saving configuration 2023-10-18T14:34:32.729 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T14:34:32.750 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:34:32.776 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:29:18.395833', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-18T14:34:32.803 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:29:18.397283', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-18T14:34:32.803 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T14:34:32.809 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-18T14:34:32.810 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2023-10-18T14:34:32.810 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T14:34:32.929 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0092ebfaf0>, signals=[15]) 2023-10-18T14:34:32.930 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T14:34:32.936 INFO:teuthology.task.internal:Opening connections... 2023-10-18T14:34:32.937 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2023-10-18T14:34:32.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:34:33.008 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2023-10-18T14:34:33.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:34:33.076 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T14:34:33.082 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2023-10-18T14:34:33.103 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2023-10-18T14:34:33.103 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2023-10-18T14:34:33.163 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:34:33.163 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="8.6 (Ootpa)" 2023-10-18T14:34:33.163 INFO:teuthology.orchestra.run.smithi026.stdout:ID="rhel" 2023-10-18T14:34:33.163 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="fedora" 2023-10-18T14:34:33.164 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="8.6" 2023-10-18T14:34:33.164 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:34:33.164 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-18T14:34:33.164 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2023-10-18T14:34:33.164 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-18T14:34:33.165 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:34:33.165 INFO:teuthology.orchestra.run.smithi026.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:34:33.165 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:34:33.165 INFO:teuthology.orchestra.run.smithi026.stdout: 2023-10-18T14:34:33.165 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:34:33.166 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-18T14:34:33.166 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:34:33.166 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-18T14:34:33.167 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2023-10-18T14:34:33.207 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2023-10-18T14:34:33.228 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2023-10-18T14:34:33.228 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2023-10-18T14:34:33.286 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="Red Hat Enterprise Linux" 2023-10-18T14:34:33.286 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="8.6 (Ootpa)" 2023-10-18T14:34:33.286 INFO:teuthology.orchestra.run.smithi179.stdout:ID="rhel" 2023-10-18T14:34:33.287 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="fedora" 2023-10-18T14:34:33.287 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="8.6" 2023-10-18T14:34:33.287 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:34:33.287 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-18T14:34:33.288 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2023-10-18T14:34:33.288 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-18T14:34:33.288 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://www.redhat.com/" 2023-10-18T14:34:33.288 INFO:teuthology.orchestra.run.smithi179.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-18T14:34:33.288 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:34:33.289 INFO:teuthology.orchestra.run.smithi179.stdout: 2023-10-18T14:34:33.289 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:34:33.289 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-18T14:34:33.289 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-18T14:34:33.289 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-18T14:34:33.290 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2023-10-18T14:34:33.315 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T14:34:33.329 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T14:34:33.385 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T14:34:33.385 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:34:33.389 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:34:33.407 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T14:34:33.435 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T14:34:33.435 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:34:33.449 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:34:33.476 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T14:34:33.549 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T14:34:33.606 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-10-18T14:34:33.607 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}} 2023-10-18T14:34:33.608 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-10-18T14:34:33.608 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:34:33.609 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:34:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:34:33.627 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2023-10-18T14:34:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:34:33.629 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2023-10-18T14:34:33.685 INFO:teuthology.orchestra.run.smithi179.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-18T14:34:33.686 INFO:teuthology.task.kernel:Running kernel on smithi179: 4.18.0-372.9.1.el8.x86_64 2023-10-18T14:34:33.686 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum install -y kernel 2023-10-18T14:34:33.688 INFO:teuthology.orchestra.run.smithi026.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-18T14:34:33.688 INFO:teuthology.task.kernel:Running kernel on smithi026: 4.18.0-372.9.1.el8.x86_64 2023-10-18T14:34:33.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2023-10-18T14:34:34.788 INFO:teuthology.orchestra.run.smithi179.stdout:Updating Subscription Management repositories. 2023-10-18T14:34:34.789 INFO:teuthology.orchestra.run.smithi179.stdout:Unable to read consumer identity 2023-10-18T14:34:34.908 INFO:teuthology.orchestra.run.smithi026.stdout:Updating Subscription Management repositories. 2023-10-18T14:34:34.909 INFO:teuthology.orchestra.run.smithi026.stdout:Unable to read consumer identity 2023-10-18T14:34:36.245 INFO:teuthology.orchestra.run.smithi179.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-10-18T14:34:36.333 INFO:teuthology.orchestra.run.smithi026.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-10-18T14:34:36.912 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2023-10-18T14:34:37.492 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-18T14:34:40.362 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 479 kB/s | 24 kB 00:00 2023-10-18T14:34:41.057 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-18T14:34:41.116 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2023-10-18T14:34:41.117 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2023-10-18T14:34:41.118 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2023-10-18T14:34:41.160 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 468 kB/s | 24 kB 00:00 2023-10-18T14:34:41.242 DEBUG:teuthology.orchestra.run.smithi179:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:34:41.571 INFO:teuthology.orchestra.run.smithi179.stdout:Updating Subscription Management repositories. 2023-10-18T14:34:41.572 INFO:teuthology.orchestra.run.smithi179.stdout:Unable to read consumer identity 2023-10-18T14:34:41.838 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:01 ago on Wed 18 Oct 2023 02:34:40 PM UTC. 2023-10-18T14:34:41.880 INFO:teuthology.orchestra.run.smithi179.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-18T14:34:41.908 INFO:teuthology.orchestra.run.smithi179.stderr:Error: No packages marked for reinstall. 2023-10-18T14:34:41.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum reinstall -y kernel || true 2023-10-18T14:34:42.028 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-18T14:34:42.085 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2023-10-18T14:34:42.086 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2023-10-18T14:34:42.086 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2023-10-18T14:34:42.212 DEBUG:teuthology.orchestra.run.smithi026:> echo no | sudo yum reinstall kernel || true 2023-10-18T14:34:42.284 INFO:teuthology.orchestra.run.smithi179.stdout:Updating Subscription Management repositories. 2023-10-18T14:34:42.285 INFO:teuthology.orchestra.run.smithi179.stdout:Unable to read consumer identity 2023-10-18T14:34:42.544 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:34:40 PM UTC. 2023-10-18T14:34:42.565 INFO:teuthology.orchestra.run.smithi026.stdout:Updating Subscription Management repositories. 2023-10-18T14:34:42.566 INFO:teuthology.orchestra.run.smithi026.stdout:Unable to read consumer identity 2023-10-18T14:34:42.586 INFO:teuthology.orchestra.run.smithi179.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-18T14:34:42.615 INFO:teuthology.orchestra.run.smithi179.stderr:Error: No packages marked for reinstall. 2023-10-18T14:34:42.662 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:34:42.742 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-18T14:34:42.742 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-18T14:34:42.743 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-18T14:34:42.743 DEBUG:teuthology.parallel:result is None 2023-10-18T14:34:42.849 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:01 ago on Wed 18 Oct 2023 02:34:41 PM UTC. 2023-10-18T14:34:42.892 INFO:teuthology.orchestra.run.smithi026.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-18T14:34:42.947 INFO:teuthology.orchestra.run.smithi026.stderr:Error: No packages marked for reinstall. 2023-10-18T14:34:42.995 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum reinstall -y kernel || true 2023-10-18T14:34:43.347 INFO:teuthology.orchestra.run.smithi026.stdout:Updating Subscription Management repositories. 2023-10-18T14:34:43.347 INFO:teuthology.orchestra.run.smithi026.stdout:Unable to read consumer identity 2023-10-18T14:34:43.621 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:02 ago on Wed 18 Oct 2023 02:34:41 PM UTC. 2023-10-18T14:34:43.665 INFO:teuthology.orchestra.run.smithi026.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-18T14:34:43.696 INFO:teuthology.orchestra.run.smithi026.stderr:Error: No packages marked for reinstall. 2023-10-18T14:34:43.743 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:34:43.843 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-18T14:34:43.843 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-18T14:34:43.844 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-18T14:34:43.844 DEBUG:teuthology.parallel:result is None 2023-10-18T14:34:43.844 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T14:34:43.852 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T14:34:43.852 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:34:43.855 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:34:43.875 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T14:34:43.938 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T14:34:43.968 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T14:34:43.969 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:34:43.971 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:34:43.998 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T14:34:44.016 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T14:34:44.016 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:34:44.030 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:34:44.058 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:34:44.076 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:34:44.083 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:34:44.101 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:34:44.103 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T14:34:44.140 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T14:34:44.140 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:34:44.143 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:34:44.172 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T14:34:44.189 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T14:34:44.191 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:34:44.211 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:34:44.231 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:34:44.320 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:34:44.377 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:34:44.460 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:34:44.504 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2023-10-18T14:34:44.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:34:44.572 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:34:44.621 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:34:44.678 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:34:44.764 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:34:44.811 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2023-10-18T14:34:44.811 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:34:44.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2023-10-18T14:34:44.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2023-10-18T14:34:44.961 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:34:44.984 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:34:45.262 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T14:34:45.271 INFO:teuthology.task.internal:Starting timer... 2023-10-18T14:34:45.271 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T14:34:45.333 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T14:34:45.372 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T14:34:45.373 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2023-10-18T14:34:45.426 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2023-10-18T14:34:45.428 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2023-10-18T14:34:45.497 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2023-10-18T14:34:45.499 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T14:34:45.499 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2023-10-18T14:34:45.522 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2023-10-18T14:34:45.522 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2023-10-18T14:34:45.545 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2023-10-18T14:34:45.546 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2023-10-18T14:34:45.546 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:34:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:34:45.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:34:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:34:45.620 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T14:34:45.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:34:45.673 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:34:45.701 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T14:34:45.712 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-18T14:34:45.712 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T14:34:45.742 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T14:34:45.743 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T14:54:13.343 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909/ansible_failures.yaml 2023-10-18T14:54:13.346 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'Status code: 504 for https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 (IP: 3.225.109.36)', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:54:13.422 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1894ffb5bef24dd2b4d7afd9c4bacd41 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'Status code: 504 for https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 (IP: 3.225.109.36)', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:54:13.425 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T14:54:13.436 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T14:54:13.437 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T14:54:13.476 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:54:13.608 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:54:13.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:54:13.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:54:13.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:54:13.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:54:13.772 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T14:54:13.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T14:54:13.809 INFO:teuthology.task.internal:Duration was 1168.538241 seconds 2023-10-18T14:54:13.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T14:54:13.842 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T14:54:13.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:54:13.845 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:54:13.893 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:54:13.894 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:54:14.263 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T14:54:14.263 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2023-10-18T14:54:14.264 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:54:14.288 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2023-10-18T14:54:14.288 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:54:14.311 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T14:54:14.311 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:54:14.330 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:54:14.391 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T14:54:14.392 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:54:14.410 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:54:14.500 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T14:54:14.509 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T14:54:14.509 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:54:14.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:54:14.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T14:54:14.590 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:54:14.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:54:14.617 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2023-10-18T14:54:14.647 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2023-10-18T14:54:14.662 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:54:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:54:14.688 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:54:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:54:14.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T14:54:14.728 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T14:54:14.729 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909/remote/smithi026 2023-10-18T14:54:14.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:54:14.795 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909/remote/smithi179 2023-10-18T14:54:14.796 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:54:14.842 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T14:54:14.843 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:54:14.846 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:54:14.893 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T14:54:14.904 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T14:54:14.904 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T14:54:14.926 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T14:54:14.926 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:54:14.929 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:54:14.943 INFO:teuthology.orchestra.run.smithi026.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:54 /home/ubuntu/cephtest 2023-10-18T14:54:14.949 INFO:teuthology.orchestra.run.smithi179.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:54 /home/ubuntu/cephtest 2023-10-18T14:54:14.950 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T14:54:14.959 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T14:54:15.026 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T14:54:15.056 DEBUG:teuthology.nuke:shortname: smithi026 2023-10-18T14:54:15.056 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:54:15.081 DEBUG:teuthology.nuke:shortname: smithi179 2023-10-18T14:54:15.081 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:54:15.086 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:29:18.395833', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-18T14:54:15.112 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430909', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:29:18.397283', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-18T14:54:15.137 INFO:teuthology.orchestra.console.smithi026:Power off 2023-10-18T14:54:15.138 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:54:15.164 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2023-10-18T14:54:15.165 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:54:15.181 INFO:teuthology.orchestra.console.smithi179:Power off 2023-10-18T14:54:15.182 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:54:15.192 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2023-10-18T14:54:15.206 DEBUG:teuthology.orchestra.console.smithi179:power off output: Chassis Power Control: Down/Off 2023-10-18T14:54:15.206 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:54:15.228 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2023-10-18T14:54:19.193 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:54:19.230 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:54:19.304 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2023-10-18T14:54:19.342 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2023-10-18T14:54:23.306 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:54:23.343 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:54:23.419 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2023-10-18T14:54:23.454 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2023-10-18T14:54:23.521 INFO:teuthology.orchestra.console.smithi026:Power off completed 2023-10-18T14:54:27.458 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:54:27.568 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is off 2023-10-18T14:54:27.669 INFO:teuthology.orchestra.console.smithi179:Power off completed 2023-10-18T14:54:27.775 INFO:teuthology.run:Summary data: description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} duration: 1168.538241147995 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Please do not file bug reports about these packages in Fedora'', ''Status code: 504 for https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 (IP: 3.225.109.36)'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1894ffb5bef24dd2b4d7afd9c4bacd41 status: dead success: false 2023-10-18T14:54:27.776 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:54:27.849 INFO:teuthology.run:DEAD