2022-05-02T14:28:35.135 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166 2022-05-02T14:28:35.135 INFO:teuthology.results:subset: '30/32' 2022-05-02T14:28:35.135 INFO:teuthology.results:seed: '6637' 2022-05-02T15:22:55.499 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166 2022-05-02T15:22:55.531 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-02T18:55:11.374 INFO:teuthology.results:Tests finished! gathering results... 2022-05-02T18:55:12.895 INFO:scrape:Found 16 jobs 2022-05-02T18:55:12.922 INFO:scrape:Found 2 distinct failure reasons 2022-05-02T18:55:12.922 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/ffsb.sh 2022-05-02T18:55:12.922 INFO:scrape:1 jobs: ['6818503'] 2022-05-02T18:55:12.922 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-02T18:55:12.922 INFO:scrape: 2022-05-02T18:55:12.922 INFO:scrape:Failure: "2022-05-02T18:21:07.354668+0000 osd.4 (osd.4) 12 : cluster [ERR] 5.18s0 deep-scrub : stat mismatch, got 0/1 objects, 0/0 clones, 0/1 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/715 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log 2022-05-02T18:55:12.923 INFO:scrape:1 jobs: ['6818496'] 2022-05-02T18:55:12.923 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-05-02T18:55:12.923 INFO:scrape: 2022-05-02T18:55:12.924 INFO:teuthology.results:Sending results to jlayton@redhat.com: Test Run: jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 14 Fail ================================================================= [6818496] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 04:02:36 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818496/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818496/ "2022-05-02T18:21:07.354668+0000 osd.4 (osd.4) 12 : cluster [ERR] 5.18s0 deep-scrub : stat mismatch, got 0/1 objects, 0/0 clones, 0/1 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/715 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log [6818503] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 03:24:36 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818503/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818503/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55571e3061db466483cf2113df8d3fb8 Command failed (workunit test suites/ffsb.sh) on smithi071 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5faec6ad8cc6050dab5d3cff07b1f32ae8f6c4b5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' Pass ================================================================= [6818489] fs:thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:07:00 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818489/ [6818490] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:50:41 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818490/ [6818491] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:22:47 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818491/ [6818492] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:27:58 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818492/ [6818493] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:13:19 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818493/ [6818494] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:21:33 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818494/ [6818495] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:24 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818495/ [6818497] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:27:59 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818497/ [6818498] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:22:14 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818498/ [6818499] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:13:14 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818499/ [6818500] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:16:37 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818500/ [6818501] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:16:23 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818501/ [6818502] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:30:10 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818502/ [6818504] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:29:48 info: http://pulpito.ceph.com/jlayton-2022-05-02_14:27:24-fs:thrash-master-testing-default-smithi/6818504/