2022-04-06T07:58:49.414 INFO:root:teuthology version: 1.1.0-3dd79763 2022-04-06T07:58:49.416 INFO:teuthology.results:subset: '21/32' 2022-04-06T07:58:49.416 INFO:teuthology.results:seed: '6329' 2022-04-06T10:05:26.884 INFO:root:teuthology version: 1.1.0-3dd79763 2022-04-06T10:05:26.912 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-06T12:52:40.212 INFO:teuthology.results:Tests finished! gathering results... 2022-04-06T12:52:50.178 INFO:scrape:Found 120 jobs 2022-04-06T12:53:00.845 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:00.846 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:00.847 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:01.080 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:01.080 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:01.081 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:01.311 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:01.312 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:01.312 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:01.542 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:01.542 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:01.543 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:01.779 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:01.779 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:01.780 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:02.005 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:02.005 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:02.006 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:02.226 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:02.227 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:02.228 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:02.462 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:02.463 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:02.463 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:02.682 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:02.683 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:02.684 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:02.912 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:02.913 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:02.913 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:03.148 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:03.149 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:03.149 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:03.386 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f6afd2bc7cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 2022-04-06T12:53:03.387 WARNING:scrape:Ignoring malparsed backtrace: "compat": {}, , "incompat": { , "feature_1": "base v0.20", 2022-04-06T12:53:03.388 WARNING:scrape:Ignoring malparsed backtrace: { , "addr": "172.21.15.83:6836", , "nonce": 3381640649, 2022-04-06T12:53:08.440 INFO:scrape:Found 12 distinct failure reasons 2022-04-06T12:53:08.441 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2022-04-06T12:53:08.442 INFO:scrape:1 jobs: ['6778854'] 2022-04-06T12:53:08.442 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2022-04-06T12:53:08.443 INFO:scrape: 2022-04-06T12:53:08.443 INFO:scrape:Failure: Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid fd5300a6-b591-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.37 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' 2022-04-06T12:53:08.444 INFO:scrape:19 jobs: ['6778954', '6778890', '6778915', '6778876', '6778873', '6778868', '6778961', '6778910', '6778855', '6778899', '6778894', '6778928', '6778904', '6778859', '6778882', '6778939', '6778947', '6778932', '6778864'] 2022-04-06T12:53:08.445 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.445 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/3', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/fs/misc}', 'workunit/fs/norstats}', 'workunit/fs/test_o_trunc}', 'workunit/kernel_untar_build}', 'workunit/suites/blogbench}', 'workunit/suites/dbench}', 'workunit/suites/ffsb}', 'workunit/suites/fsstress}', 'workunit/suites/fsx}', 'workunit/suites/fsync-tester}', 'workunit/suites/iogen}', 'workunit/suites/pjd}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-04-06T12:53:08.446 INFO:scrape: 2022-04-06T12:53:08.446 INFO:scrape:Failure: "2022-04-06T10:31:11.064292+0000 mon.a (mon.0) 480 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-04-06T12:53:08.447 INFO:scrape:2 jobs: ['6778964', '6778936'] 2022-04-06T12:53:08.447 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.448 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/metrics}', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.448 INFO:scrape: 2022-04-06T12:53:08.448 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-04-06T12:53:08.449 INFO:scrape:3 jobs: ['6778943', '6778952', '6778945'] 2022-04-06T12:53:08.449 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'prefetch_entire_dirfrags/no', 'races', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.450 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.451 INFO:scrape: 2022-04-06T12:53:08.451 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2022-04-06T12:53:08.452 INFO:scrape:6 jobs: ['6778946', '6778926', '6778941', '6778888', '6778914', '6778889'] 2022-04-06T12:53:08.452 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.453 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.454 INFO:scrape: 2022-04-06T12:53:08.454 INFO:scrape:Failure: Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 33f12728-b58d-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' 2022-04-06T12:53:08.455 INFO:scrape:2 jobs: ['6778921', '6778846'] 2022-04-06T12:53:08.455 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.456 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/3', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/dbench}', 'workunit/suites/ffsb}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-04-06T12:53:08.456 INFO:scrape: 2022-04-06T12:53:08.457 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1649232640.868:192): avc: denied { node_bind } for pid=1489 comm="ping" saddr=172.21.15.27 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2022-04-06T12:53:08.457 INFO:scrape:2 jobs: ['6778847', '6778906'] 2022-04-06T12:53:08.458 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'mds', 'mon', 'mount', 'osd}', 'ranks/1', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.458 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{mon-debug', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/dbench', 'tasks/{1-thrash/mon', 'thrashosds-health', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.459 INFO:scrape: 2022-04-06T12:53:08.459 INFO:scrape:Failure: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' 2022-04-06T12:53:08.460 INFO:scrape:2 jobs: ['6778858', '6778879'] 2022-04-06T12:53:08.460 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.461 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.461 INFO:scrape: 2022-04-06T12:53:08.462 INFO:scrape:Failure: "2022-04-06T09:00:53.294977+0000 mds.d (mds.0) 1 : cluster [WRN] client.4788 isn't responding to mclientcaps(revoke), ino 0x100000018b4 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.014712 seconds ago" in cluster log 2022-04-06T12:53:08.462 INFO:scrape:1 jobs: ['6778869'] 2022-04-06T12:53:08.463 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.463 INFO:scrape: 2022-04-06T12:53:08.464 INFO:scrape:Crash: "2022-04-06T09:55:45.327190+0000 mon.a (mon.0) 2495 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log 2022-04-06T12:53:08.465 INFO:scrape:ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0) [0x7f6afcff73c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7f6afd2bc834] 5: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f6afd2bc9de] 6: (MDCache::dispatch_fragment_dir(boost::intrusive_ptr&)+0x1701) [0x562d407aa181] 7: (MDCache::fragment_frozen(boost::intrusive_ptr&, int)+0xf3) [0x562d407aa343] 8: (MDCache::fragment_mark_and_complete(boost::intrusive_ptr&)+0xe16) [0x562d407ab366] 9: (MDCache::merge_dir(CInode*, frag_t)+0x496) [0x562d407ac266] 10: ceph-mds(+0x45b6a9) [0x562d408b46a9] 11: (Context::complete(int)+0xd) [0x562d4062d78d] 12: (CommonSafeTimer::timer_thread()+0x15d) [0x7f6afd3c91ad] 13: (CommonSafeTimerThread::entry()+0x11) [0x7f6afd3ca6b1] 14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6afcfeb609] 15: clone() 2022-04-06T12:53:08.465 INFO:scrape:1 jobs: ['6778866'] 2022-04-06T12:53:08.466 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.466 INFO:scrape: 2022-04-06T12:53:08.467 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2022-04-06T12:53:08.467 INFO:scrape:1 jobs: ['6778886'] 2022-04-06T12:53:08.468 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/acls}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.468 INFO:scrape: 2022-04-06T12:53:08.469 INFO:scrape:Failure: SSH connection to smithi142 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2022-04-06T12:53:08.469 INFO:scrape:2 jobs: ['6778860', '6778884'] 2022-04-06T12:53:08.470 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'races', 'ranks/1', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.470 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-06T12:53:08.471 INFO:scrape: 2022-04-06T12:53:08.477 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/ failed: 40 dead: 2 running: 0 waiting: 0 queued: 0 passed: 78 Fail ================================================================= [6778846] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:43 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778846/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778846/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3825e253d3641dcb7b0797f688c8573 Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 2c13310a-b582-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.8 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778847] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:13:34 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778847/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778847/ SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1649232640.868:192): avc: denied { node_bind } for pid=1489 comm="ping" saddr=172.21.15.27 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6778854] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:16:38 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778854/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd751d610d80442297e5bdfc686ee1f6 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [6778855] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} ----------------------------------------------------------------- time: 00:10:42 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778855/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778855/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3be582d50d24f3da6e3bec9ffc807be Command failed on smithi082 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 1416192c-b583-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.82 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778858] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 04:15:41 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778858/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778858/ sentry: https://sentry.ceph.com/organizations/ceph/?query=92cd27e653fc43b9a26f58e1fac112f4 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6778859] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} ----------------------------------------------------------------- time: 00:10:36 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778859/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83e9f4f5bdc74b72b1a21ec7402cb5fe Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 956e18bc-b583-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.57 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778864] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} ----------------------------------------------------------------- time: 00:10:42 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778864/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f614f1839224befb7f449550af827b0 Command failed on smithi107 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 0c318cf4-b584-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.107 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778866] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 02:25:23 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778866/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778866/ "2022-04-06T09:55:45.327190+0000 mon.a (mon.0) 2495 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log [6778868] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:16 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778868/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=397b00bd9ff74c259e848affa27732cc Command failed on smithi070 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 6fa2c5f0-b584-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.70 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778869] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 04:27:56 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778869/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778869/ "2022-04-06T09:00:53.294977+0000 mds.d (mds.0) 1 : cluster [WRN] client.4788 isn't responding to mclientcaps(revoke), ino 0x100000018b4 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.014712 seconds ago" in cluster log [6778873] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:51 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778873/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778873/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb7b24158a4f45ceaa94bf2d4d48a2b3 Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 2b480374-b585-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.8 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778876] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} ----------------------------------------------------------------- time: 00:10:44 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778876/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778876/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4a36ecfa54eb4d6a83743b5c5c307707 Command failed on smithi061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid fc97f2f4-b585-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.61 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778879] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 03:17:00 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778879/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778879/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7dc30124e9454707bb145b99b155a695 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi115 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6778882] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} ----------------------------------------------------------------- time: 00:10:31 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778882/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778882/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac0c23a222aa4931987bfae7f596bc74 Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 7e725904-b586-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.57 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778886] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} ----------------------------------------------------------------- time: 00:11:41 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778886/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778886/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a842044113b642e892ad88058a4beaad Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) [6778888] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:33:31 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778888/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778888/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb487703a52946a29cb6c42e6c522607 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi074 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6778889] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:31:16 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778889/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778889/ sentry: https://sentry.ceph.com/organizations/ceph/?query=adb243a3eab94cafb880da221baea9bf Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6778890] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 00:11:02 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778890/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778890/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29aca3f796824f3db8636fa2147b71dc Command failed on smithi018 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid acb746c0-b587-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.18 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778894] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:11:09 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778894/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f84195a4aec4a55b85a7f81e2a5e928 Command failed on smithi025 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid eebfcd98-b588-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.25 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778899] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} ----------------------------------------------------------------- time: 00:11:13 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778899/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3cc1f58a632b40229d21a44d4f46af0d Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 6a77e2ae-b589-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778904] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} ----------------------------------------------------------------- time: 00:11:00 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778904/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778904/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2f19c14188b46dfa4e6606b888cca6b Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 21278252-b58a-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778906] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:50:16 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778906/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778906/ SELinux denials found on ubuntu@smithi036.front.sepia.ceph.com: ['type=AVC msg=audit(1649235892.168:193): avc: denied { node_bind } for pid=1448 comm="ping" saddr=172.21.15.36 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6778910] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:13 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778910/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778910/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75a86907711e48149cea0a3fcaee7d3d Command failed on smithi073 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid f20eb9e4-b58a-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.73 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778914] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:30:05 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778914/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778914/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2952415aca6a497ebf7b5df4d8b4e3e7 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi008 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6778915] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} ----------------------------------------------------------------- time: 00:13:12 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778915/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778915/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c414547f05fc49ee867fe50508e5c450 Command failed on smithi002 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 3925c386-b58b-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778921] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} ----------------------------------------------------------------- time: 00:11:18 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778921/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778921/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b0024b19b25456db19887f71cd7187d Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 33f12728-b58d-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778926] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:34:54 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778926/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2045326244124119a6ecf1bff2224983 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6778928] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:31 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778928/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778928/ sentry: https://sentry.ceph.com/organizations/ceph/?query=05d6384cec9c4e8bb7ece3bbf840ea07 Command failed on smithi070 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 7e8f997c-b58d-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.70 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778932] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} ----------------------------------------------------------------- time: 00:10:37 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778932/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778932/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f3d484138a8404ab2fe4229536d58f6 Command failed on smithi061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 7b74216c-b58e-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.61 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778936] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} ----------------------------------------------------------------- time: 00:22:57 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778936/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778936/ "2022-04-06T09:50:04.311977+0000 mon.a (mon.0) 367 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6778939] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:32 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778939/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778939/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73275869c33a47ac9729d5b4d3336ab5 Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 1d4b2184-b58f-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.37 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778941] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:33:03 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778941/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e86000bebb445eaa0156b6b22dd9fcc Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi049 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6778943] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:18:38 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778943/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778943/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a9dd3059f17d49369010f662e23ab7ed reached maximum tries (90) after waiting for 540 seconds [6778945] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:18:22 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778945/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778945/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0709c1c1f9949cf900bd590c5c0aff5 reached maximum tries (90) after waiting for 540 seconds [6778946] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:15:47 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778946/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778946/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0fd6be3e17ae4f44912bb555706f8bcc Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6778947] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:10:15 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778947/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778947/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c68b6d162444786b6d92d9822cc0aa6 Command failed on smithi070 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid 7a103674-b590-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.70 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778952] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:17:54 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778952/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778952/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee47f930f1bc4b4bb7557e12d5189125 reached maximum tries (90) after waiting for 540 seconds [6778954] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:45 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778954/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778954/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1cf641eeefa34be0acd340e30d30df0f Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid fd5300a6-b591-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.37 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778961] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} ----------------------------------------------------------------- time: 00:10:54 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778961/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778961/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15e4beb3e56b4b468d6257d3d1de6cbb Command failed on smithi067 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:cfb8f943163b374162da0d7b0240f267dd46e4e1 -v bootstrap --fsid b2e498a8-b592-11ec-8c36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial- daemons --skip-monitoring-stack --mon-ip 172.21.15.67 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' [6778964] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:44:25 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778964/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778964/ "2022-04-06T10:31:11.064292+0000 mon.a (mon.0) 480 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log Dead ================================================================= [6778860] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 01:27:21 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778860/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778860/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f7341c7a9504fd589f3eba7c30e847e SSH connection to smithi142 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [6778884] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:30:57 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778884/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778884/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ceff374c5824970bc497de2fb2e1c30 SSH connection to smithi139 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' Pass ================================================================= [6778848] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:18:52 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778848/ [6778849] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 00:56:52 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778849/ [6778850] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} time: 00:20:57 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778850/ [6778851] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:42:19 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778851/ [6778852] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} time: 00:48:29 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778852/ [6778853] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:23:01 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778853/ [6778856] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:35:39 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778856/ [6778857] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:00 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778857/ [6778861] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 01:01:41 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778861/ [6778862] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:55:37 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778862/ [6778863] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:07:26 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778863/ [6778865] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:24:33 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778865/ [6778867] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:16:22 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778867/ [6778870] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:32:45 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778870/ [6778871] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:56 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778871/ [6778872] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:18:14 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778872/ [6778874] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:22:31 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778874/ [6778875] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:24:13 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778875/ [6778877] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:25 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778877/ [6778878] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:23:43 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778878/ [6778880] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:43:36 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778880/ [6778881] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:24:14 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778881/ [6778883] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} time: 00:23:34 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778883/ [6778885] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:36 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778885/ [6778887] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:36:11 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778887/ [6778891] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:17:07 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778891/ [6778892] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 01:42:51 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778892/ [6778893] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:09:48 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778893/ [6778895] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:39 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778895/ [6778896] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:59:35 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778896/ [6778897] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 00:55:43 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778897/ [6778898] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:22:28 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778898/ [6778900] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:34:46 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778900/ [6778901] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} time: 00:28:33 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778901/ [6778902] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:33 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778902/ [6778903] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:26:12 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778903/ [6778905] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/whitelist_health} time: 00:15:17 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778905/ [6778907] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 01:15:54 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778907/ [6778908] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:24:12 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778908/ [6778909] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:03:41 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778909/ [6778911] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:12:56 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778911/ [6778912] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:24:47 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778912/ [6778913] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:07:16 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778913/ [6778916] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:07:53 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778916/ [6778917] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:29 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778917/ [6778918] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} time: 00:39:12 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778918/ [6778919] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:11:15 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778919/ [6778920] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:01:03 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778920/ [6778922] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:16:12 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778922/ [6778923] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:23:12 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778923/ [6778924] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:14:43 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778924/ [6778925] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} time: 00:15:17 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778925/ [6778927] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:36 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778927/ [6778929] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:24:35 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778929/ [6778930] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:58:14 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778930/ [6778931] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:23:06 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778931/ [6778933] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:02:47 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778933/ [6778934] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} time: 00:24:24 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778934/ [6778935] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python} time: 00:07:49 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778935/ [6778937] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:04 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778937/ [6778938] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:15:42 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778938/ [6778940] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} time: 00:13:26 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778940/ [6778942] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} time: 01:02:17 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778942/ [6778944] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:23:56 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778944/ [6778948] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:49:37 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778948/ [6778949] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:25 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778949/ [6778950] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:11:22 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778950/ [6778951] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:11:41 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778951/ [6778953] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 02:39:01 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778953/ [6778955] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:16:19 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778955/ [6778956] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:12:59 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778956/ [6778957] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} time: 00:25:14 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778957/ [6778958] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:54:32 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778958/ [6778959] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:11:39 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778959/ [6778960] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:54 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778960/ [6778962] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:23:47 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778962/ [6778963] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} time: 00:31:10 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778963/ [6778965] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:22:51 info: http://pulpito.ceph.com/vshankar-2022-04-06_07:58:13-fs-wip-vshankar-testing-20220403-170344-distro-default-smithi/6778965/