2022-02-23T16:52:08.609 INFO:root:teuthology version: 1.1.0-6b607b46
2022-02-23T16:52:08.610 INFO:teuthology.results:subset: '111/120000'
2022-02-23T16:52:08.611 INFO:teuthology.results:seed: '7511'
2022-02-24T03:55:59.977 INFO:root:teuthology version: 1.1.0-6b607b46
2022-02-24T03:56:00.009 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-02-24T07:18:15.365 INFO:teuthology.results:Tests finished! gathering results...
2022-02-24T07:18:21.605 INFO:scrape:Found 61 jobs
2022-02-24T07:18:43.574 INFO:scrape:Found 9 distinct failure reasons
2022-02-24T07:18:43.575 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi111.front.sepia.ceph.com: ['type=AVC msg=audit(1645640151.823:10689): avc:  denied  { sendto } for  pid=26392 comm="chronyd" path="/run/chrony/chronyc.147527.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:NetworkManager_dispatcher_t:s0 tclass=unix_dgram_socket permissive=1', 'type=AVC msg=audit(1645639445.672:7679): avc:  denied  { sendto } for  pid=26392 comm="chronyd" path="/run/chrony/chronyc.89008.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:NetworkManager_dispatcher_t:s0 tclass=unix_dgram_socket permissive=1']
2022-02-24T07:18:43.576 INFO:scrape:44 jobs
2022-02-24T07:18:43.577 INFO:scrape:suites intersection: []
2022-02-24T07:18:43.578 INFO:scrape:suites union: ['1-ceph', '1-client', '1-mount', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-upgrade', '2-workload/dbench', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/pacific}}', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-compat_client', '4-verify}', '4-verify}}', '5-client-sanity}}', 'centos_8.stream_container_tools', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1-node-1-mds-1-osd', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/centos_latest}', 'distro/{centos_8}', 'fs/32bits/{begin/{0-install', 'fs/cephadm/multivolume/{0-start', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{mon-debug', 'overrides/{multimds/no', 'overrides/{multimds/yes', 'overrides/{no_client_pidfile', 'overrides/{pg-warn', 'overrides/{whitelist_health', 'pg-warn', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'roles', 'session_timeout', 'syntax', 'tasks/admin}', 'tasks/asok_dump_tree}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/dbench', 'tasks/forward-scrub}', 'tasks/fragment}', 'tasks/fsstress', 'tasks/mgr-osd-full}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'tasks/{0-octopus', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/clone}}', 'thrashosds-health', 'validater/lockdep}', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.578 INFO:scrape:
2022-02-24T07:18:43.579 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/ffsb.sh
2022-02-24T07:18:43.579 INFO:scrape:1 jobs: ['6702633']
2022-02-24T07:18:43.580 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.580 INFO:scrape:
2022-02-24T07:18:43.581 INFO:scrape:Crash: SELinux denials found on ubuntu@smithi041.front.sepia.ceph.com: ['type=AVC msg=audit(1645639831.635:8200): avc:  denied  { sendto } for  pid=21479 comm="chronyd" path="/run/chrony/chronyc.40833.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:NetworkManager_dispatcher_t:s0 tclass=unix_dgram_socket permissive=1', 'type=AVC msg=audit(1645640558.870:16662): avc:  denied  { sendto } for  pid=21479 comm="chronyd" path="/run/chrony/chronyc.90766.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:NetworkManager_dispatcher_t:s0 tclass=unix_dgram_socket permissive=1']
2022-02-24T07:18:43.581 INFO:scrape:ceph version 17.0.0-10430-g4fba29ce (4fba29ce98c0f535f72d6211e12a92b0f5cc66df) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12c20) [0x7f0feb94fc20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f0fec96e0a2]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x283265) [0x7f0fec96e265]
 6: (MDSTableClient::handle_request(boost::intrusive_ptr<MMDSTableRequest const> const&)+0x12b0) [0x555eab0a53a0]
 7: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xfed) [0x555eaadd617d]
 8: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7cb) [0x555eaadd8b1b]
 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x555eaadd913c]
 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x108) [0x555eaadc8038]
 11: (DispatchQueue::entry()+0x14fa) [0x7f0fecbf402a]
 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0feccaa731]
 13: /lib64/libpthread.so.0(+0x817f) [0x7f0feb94517f]
 14: clone()
2022-02-24T07:18:43.582 INFO:scrape:1 jobs: ['6702625']
2022-02-24T07:18:43.583 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.583 INFO:scrape:
2022-02-24T07:18:43.584 INFO:scrape:Failure: "2022-02-23T17:34:18.383359+0000 osd.4 (osd.4) 20 : cluster [ERR] 5.1ds0 deep-scrub : stat mismatch, got 0/2 objects, 0/0 clones, 0/2 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/8388608 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
2022-02-24T07:18:43.584 INFO:scrape:1 jobs: ['6702596']
2022-02-24T07:18:43.585 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.585 INFO:scrape:
2022-02-24T07:18:43.586 INFO:scrape:Failure: "2022-02-23T18:04:20.271169+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022-02-24T07:18:43.586 INFO:scrape:1 jobs: ['6702609']
2022-02-24T07:18:43.587 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/metrics}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.587 INFO:scrape:
2022-02-24T07:18:43.587 INFO:scrape:Failure: Test failure: test_subvolume_no_upgrade_v1_sanity (tasks.cephfs.test_volumes.TestMisc)
2022-02-24T07:18:43.588 INFO:scrape:2 jobs: ['6702593', '6702634']
2022-02-24T07:18:43.588 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.589 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.589 INFO:scrape:
2022-02-24T07:18:43.590 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi111.front.sepia.ceph.com: ['type=AVC msg=audit(1645637958.294:14311): avc:  denied  { sendto } for  pid=21526 comm="chronyd" path="/run/chrony/chronyc.78432.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:NetworkManager_dispatcher_t:s0 tclass=unix_dgram_socket permissive=1', 'type=AVC msg=audit(1645637314.980:8199): avc:  denied  { sendto } for  pid=21526 comm="chronyd" path="/run/chrony/chronyc.40962.sock" scontext=system_u:system_r:chronyd_t:s0 tcontext=system_u:system_r:NetworkManager_dispatcher_t:s0 tclass=unix_dgram_socket permissive=1']
2022-02-24T07:18:43.590 INFO:scrape:3 jobs: ['6702595', '6702584', '6702608']
2022-02-24T07:18:43.591 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.591 INFO:scrape:suites union: ['1-ceph', '1-client', '2-logrotate}', '2-upgrade', '3-compat_client/no}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{multimds/yes', 'overrides/{no_client_pidfile', 'pg-warn', 'tasks/backtrace}', 'tasks/damage}', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.592 INFO:scrape:
2022-02-24T07:18:43.592 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2022-02-24T07:18:43.593 INFO:scrape:1 jobs: ['6702612']
2022-02-24T07:18:43.593 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-02-24T07:18:43.594 INFO:scrape:
2022-02-24T07:18:43.594 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring)
2022-02-24T07:18:43.595 INFO:scrape:1 jobs: ['6702592']
2022-02-24T07:18:43.595 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2022-02-24T07:18:43.596 INFO:scrape: