2021-08-03T09:08:40.021 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-08-03T09:08:40.022 INFO:teuthology.results:subset: '3/4' 2021-08-03T09:08:40.023 INFO:teuthology.results:seed: '8378' 2021-08-03T11:45:59.026 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-08-03T11:45:59.064 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-08-03T12:16:01.627 INFO:teuthology.results:Tests finished! gathering results... 2021-08-03T12:16:16.308 INFO:scrape:Found 216 jobs 2021-08-03T12:16:59.124 INFO:scrape:Found 8 distinct failure reasons 2021-08-03T12:16:59.126 INFO:scrape:Failure: '2021-08-03T11:28:55.954859+00:00 smithi119 kernel: WARNING: CPU: 0 PID: 42319 at fs/ceph/mds_client.c:4495 check_session_state+0x55/0x60 [ceph] ' in syslog 2021-08-03T12:16:59.126 INFO:scrape:94 jobs 2021-08-03T12:16:59.128 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.128 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8.stream}', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'rhel_8}', 'tasks/acls}', 'tasks/admin}', 'tasks/alternate-pool}', 'tasks/auto-repair}', 'tasks/backtrace}', 'tasks/cap-flush}', 'tasks/client-readahead}', 'tasks/client-recovery}', 'tasks/damage}', 'tasks/exports}', 'tasks/forward-scrub}', 'tasks/fragment}', 'tasks/journal-repair}', 'tasks/mds-flush}', 'tasks/multimds_misc}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/scrub}', 'tasks/sessionmap}', 'tasks/snap-schedule}', 'tasks/strays}', 'tasks/test_journal_migration}', 'tasks/volumes/{overrides', 'test/basic}}', 'test/clone}}', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.129 INFO:scrape: 2021-08-03T12:16:59.129 INFO:scrape:Failure: Test failure: test_ephemeral_random_failover (tasks.cephfs.test_exports.TestEphemeralPins) 2021-08-03T12:16:59.130 INFO:scrape:1 jobs: ['6309140'] 2021-08-03T12:16:59.130 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8.stream}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.131 INFO:scrape: 2021-08-03T12:16:59.131 INFO:scrape:Failure: Test failure: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2021-08-03T12:16:59.131 INFO:scrape:1 jobs: ['6309190'] 2021-08-03T12:16:59.132 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.132 INFO:scrape: 2021-08-03T12:16:59.133 INFO:scrape:Failure: No module named 'tasks.ceph' 2021-08-03T12:16:59.134 INFO:scrape:8 jobs: ['6309202', '6309185', '6309186', '6309187', '6309200', '6309201', '6309205', '6309204'] 2021-08-03T12:16:59.134 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'mds', 'mon', 'no_client_pidfile', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.135 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/sessionmap}', 'tasks/truncate_delay}', 'tasks/workunit/quota}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.135 INFO:scrape: 2021-08-03T12:16:59.135 INFO:scrape:Failure: Command failed on smithi173 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 2021-08-03T12:16:59.136 INFO:scrape:3 jobs: ['6309199', '6309198', '6309197'] 2021-08-03T12:16:59.136 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/test_journal_migration}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.137 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/test_journal_migration}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.137 INFO:scrape: 2021-08-03T12:16:59.138 INFO:scrape:Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) 2021-08-03T12:16:59.139 INFO:scrape:6 jobs: ['6309160', '6309050', '6309051', '6309159', '6309052', '6309158'] 2021-08-03T12:16:59.139 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.139 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.140 INFO:scrape: 2021-08-03T12:16:59.140 INFO:scrape:Crash: '2021-08-03T10:57:20.542653+00:00 smithi144 kernel: WARNING: CPU: 7 PID: 39112 at fs/ceph/mds_client.c:4456 check_session_state+0x4e/0x60 [ceph] ' in syslog 2021-08-03T12:16:59.141 INFO:scrape:ceph version 17.0.0-6107-ge1ccf367 (e1ccf367dff0b9d57bd5c50477cfa60c76c64762) quincy (dev) 1: /lib64/libpthread.so.0(+0x12b20) [0x7fdcaffa9b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fdcb1209048] 5: /usr/lib64/ceph/libceph-common.so.2(+0x28220b) [0x7fdcb120920b] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x128b) [0x56264235f8eb] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe1c) [0x5626420a23ac] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x7bb) [0x5626420a47bb] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x55) [0x5626420a4db5] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x128) [0x562642094b58] 11: (DispatchQueue::entry()+0x14ca) [0x7fdcb147859a] 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdcb1526ba1] 13: (Thread::_entry_func(void*)+0xd) [0x7fdcb1302bdd] 14: /lib64/libpthread.so.0(+0x814a) [0x7fdcaff9f14a] 15: clone() 2021-08-03T12:16:59.141 INFO:scrape:3 jobs: ['6309084', '6309083', '6309085'] 2021-08-03T12:16:59.142 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.142 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.143 INFO:scrape: 2021-08-03T12:16:59.143 INFO:scrape:Failure: Test failure: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2021-08-03T12:16:59.144 INFO:scrape:1 jobs: ['6309188'] 2021-08-03T12:16:59.144 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2021-08-03T12:16:59.145 INFO:scrape: