2021-02-25T17:26:13.291 INFO:root:teuthology version: 1.1.0-9a6b849c
2021-02-25T17:26:13.292 INFO:teuthology.results:seed: '7375'
2021-02-25T20:48:21.266 INFO:root:teuthology version: 1.1.0-9a6b849c
2021-02-25T20:48:21.290 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-02-26T08:48:43.169 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2021-02-26T08:52:13.432 INFO:scrape:Found 256 jobs
2021-02-26T08:52:56.501 INFO:scrape:Found 11 distinct failure reasons
2021-02-26T08:52:56.505 INFO:scrape:Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage)
2021-02-26T08:52:56.506 INFO:scrape:8 jobs: ['5902097', '5902065', '5901937', '5902129', '5902033', '5901969', '5902001', '5902161']
2021-02-26T08:52:56.506 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'no_client_pidfile.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/damage.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.508 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{rhel_8.yaml}', 'distro/{ubuntu_latest.yaml}', 'fs:functional/{begin.yaml', 'k-testing.yaml}', 'mds.yaml', 'mon.yaml', 'mount/fuse.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{distro/stock/{k-stock.yaml', 'overrides/{distro/testing/{flavor/centos_latest.yaml', 'overrides/{distro/testing/{flavor/ubuntu_latest.yaml', 'overrides/{frag_enable.yaml', 'rhel_8.yaml}', 'tasks/damage.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.508 INFO:scrape:
2021-02-26T08:52:56.509 INFO:scrape:Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)
2021-02-26T08:52:56.509 INFO:scrape:6 jobs: ['5901944', '5902008', '5902136', '5902168', '5902040', '5902072']
2021-02-26T08:52:56.510 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/mds-full.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.510 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'distro/{rhel_8.yaml}', 'distro/{ubuntu_latest.yaml}', 'fs:functional/{begin.yaml', 'k-testing.yaml}', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{distro/stock/{k-stock.yaml', 'overrides/{distro/testing/{flavor/centos_latest.yaml', 'overrides/{distro/testing/{flavor/ubuntu_latest.yaml', 'overrides/{frag_enable.yaml', 'rhel_8.yaml}', 'tasks/mds-full.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.510 INFO:scrape:
2021-02-26T08:52:56.511 INFO:scrape:Failure: Test failure: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules)
2021-02-26T08:52:56.511 INFO:scrape:1 jobs: ['5902049']
2021-02-26T08:52:56.512 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{rhel_8.yaml}', 'fs:functional/{begin.yaml', 'k-testing.yaml}', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'osd.yaml}', 'overrides/{distro/testing/{flavor/ubuntu_latest.yaml', 'overrides/{frag_enable.yaml', 'tasks/snap-schedule.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.512 INFO:scrape:
2021-02-26T08:52:56.513 INFO:scrape:Dead: b'2021-02-26T07:39:59.328 DEBUG:teuthology.exit:Finished running handlers'
2021-02-26T08:52:56.513 INFO:scrape:12 jobs: ['5902083', '5901988', '5902178', '5902063', '5902034', '5902116', '5901955', '5902054', '5902162', '5902050', '5901968', '5901935']
2021-02-26T08:52:56.514 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/fuse.yaml', 'no_client_pidfile.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.514 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'distro/{rhel_8.yaml}', 'distro/{ubuntu_latest.yaml}', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/fuse.yaml', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/client-readahead.yaml}', 'tasks/client-recovery.yaml}', 'tasks/data-scan.yaml}', 'tasks/snapshots.yaml}', 'tasks/strays.yaml}', 'tasks/test_journal_migration.yaml}', 'tasks/workunit/quota.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.515 INFO:scrape:
2021-02-26T08:52:56.515 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3eb91c4c2143df1c6d58c27e895cd61fe615d956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh'
2021-02-26T08:52:56.516 INFO:scrape:2 jobs: ['5902022', '5902150']
2021-02-26T08:52:56.516 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'osd.yaml}', 'overrides/{distro/stock/{k-stock.yaml', 'overrides/{frag_enable.yaml', 'rhel_8.yaml}', 'tasks/workunit/quota.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.516 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'distro/{ubuntu_latest.yaml}', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{distro/stock/{k-stock.yaml', 'overrides/{frag_enable.yaml', 'rhel_8.yaml}', 'tasks/workunit/quota.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.517 INFO:scrape:
2021-02-26T08:52:56.517 INFO:scrape:Crash: Test failure: test_snapclient_cache (tasks.cephfs.test_snapshots.TestSnapshots), test_snapclient_cache (tasks.cephfs.test_snapshots.TestSnapshots)
2021-02-26T08:52:56.518 INFO:scrape:ceph version 17.0.0-967-g17a14d9c (17a14d9cd44df35132113391ca026ae726484962) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7fa16bd62b20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fa16cfb58fb]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x275ac4) [0x7fa16cfb5ac4]
 6: (MDSTableClient::handle_request(boost::intrusive_ptr<MMDSTableRequest const> const&)+0x128b) [0x563399f4bd9b]
 7: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xe1c) [0x563399c8dc7c]
 8: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7bb) [0x563399c9008b]
 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x55) [0x563399c90685]
 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x128) [0x563399c80648]
 11: (DispatchQueue::entry()+0x126a) [0x7fa16d1ef0da]
 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa16d29c9a1]
 13: (Thread::_entry_func(void*)+0xd) [0x7fa16d09d2fd]
 14: /lib64/libpthread.so.0(+0x814a) [0x7fa16bd5814a]
 15: clone()
2021-02-26T08:52:56.518 INFO:scrape:6 jobs: ['5902018', '5902114', '5902082', '5902146', '5901986', '5901954']
2021-02-26T08:52:56.519 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/snapshots.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.519 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'distro/{rhel_8.yaml}', 'distro/{ubuntu_latest.yaml}', 'fs:functional/{begin.yaml', 'k-testing.yaml}', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{distro/stock/{k-stock.yaml', 'overrides/{distro/testing/{flavor/centos_latest.yaml', 'overrides/{distro/testing/{flavor/ubuntu_latest.yaml', 'overrides/{frag_enable.yaml', 'rhel_8.yaml}', 'tasks/snapshots.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.520 INFO:scrape:
2021-02-26T08:52:56.520 INFO:scrape:Failure: "2021-02-25T18:39:06.450472+0000 mon.a (mon.0) 335 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.d" in cluster log
2021-02-26T08:52:56.520 INFO:scrape:1 jobs: ['5901999']
2021-02-26T08:52:56.521 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'fs:functional/{begin.yaml', 'k-testing.yaml}', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'osd.yaml}', 'overrides/{distro/testing/{flavor/centos_latest.yaml', 'overrides/{frag_enable.yaml', 'tasks/client-readahead.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.521 INFO:scrape:
2021-02-26T08:52:56.522 INFO:scrape:Dead: b'2021-02-26T08:52:23.533 DEBUG:teuthology.orchestra.run.smithi073:> sudo logrotate /etc/logrotate.d/ceph-test.conf'
2021-02-26T08:52:56.522 INFO:scrape:1 jobs: ['5902182']
2021-02-26T08:52:56.523 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{rhel_8.yaml}', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/fuse.yaml', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/workunit/quota.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.523 INFO:scrape:
2021-02-26T08:52:56.524 INFO:scrape:Failure: Test failure: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules)
2021-02-26T08:52:56.524 INFO:scrape:2 jobs: ['5902177', '5902081']
2021-02-26T08:52:56.524 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'fs:functional/{begin.yaml', 'k-testing.yaml}', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/snap-schedule.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.525 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'fs:functional/{begin.yaml', 'k-testing.yaml}', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{distro/testing/{flavor/centos_latest.yaml', 'overrides/{distro/testing/{flavor/ubuntu_latest.yaml', 'overrides/{frag_enable.yaml', 'tasks/snap-schedule.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.525 INFO:scrape:
2021-02-26T08:52:56.526 INFO:scrape:Failure: Test failure: test_version_splitting (tasks.cephfs.test_sessionmap.TestSessionMap)
2021-02-26T08:52:56.526 INFO:scrape:1 jobs: ['5902112']
2021-02-26T08:52:56.527 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{centos_8.yaml}', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/fuse.yaml', 'no_client_pidfile.yaml', 'objectstore/bluestore-bitmap.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/sessionmap.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.527 INFO:scrape:
2021-02-26T08:52:56.527 INFO:scrape:Failure: Test failure: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules)
2021-02-26T08:52:56.528 INFO:scrape:1 jobs: ['5902113']
2021-02-26T08:52:56.528 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client.yaml', 'conf/{client.yaml', 'distro/{rhel_8.yaml}', 'fs:functional/{begin.yaml', 'mds.yaml', 'mon.yaml', 'mount/kclient/{mount.yaml', 'ms-die-on-skipped.yaml}}', 'no_client_pidfile.yaml', 'objectstore/bluestore-ec-root.yaml', 'osd.yaml}', 'overrides/{distro/stock/{k-stock.yaml', 'overrides/{frag_enable.yaml', 'rhel_8.yaml}', 'tasks/snap-schedule.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
2021-02-26T08:52:56.529 INFO:scrape: