2022-07-06T17:59:21.618 INFO:root:teuthology version: 1.1.1.dev381+g10062088 2022-07-06T17:59:21.620 INFO:teuthology.results:subset: '5/32' 2022-07-06T17:59:21.620 INFO:teuthology.results:seed: '1676' 2022-07-07T06:57:02.769 INFO:root:teuthology version: 1.1.1.dev381+g10062088 2022-07-07T06:57:02.808 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-07T18:57:29.175 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-07-07T18:58:45.371 INFO:scrape:Found 451 jobs 2022-07-07T19:00:35.094 INFO:scrape:Found 22 distinct failure reasons 2022-07-07T19:00:35.097 INFO:scrape:Dead: b'2022-07-07T16:52:10.384 DEBUG:teuthology.exit:Finished running handlers' 2022-07-07T19:00:35.098 INFO:scrape:5 jobs: ['6914794', '6914955', '6914325', '6914640', '6914551'] 2022-07-07T19:00:35.099 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/multi/{export-check', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/blogbench}}', 'wsync/no}'] 2022-07-07T19:00:35.101 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{no-subvolume}', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-no-extra-options}', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/suites/blogbench}}', 'wsync/no}'] 2022-07-07T19:00:35.102 INFO:scrape: 2022-07-07T19:00:35.103 INFO:scrape:Failure: Command failed on smithi098 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:a007ff09098d94fc80260aff6116276eb25fc941 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 29975be2-fd6a-11ec-842d-001a4aab830c -- ceph orch daemon add osd smithi098:vg_nvme/lv_3' 2022-07-07T19:00:35.104 INFO:scrape:5 jobs: ['6914419', '6914495', '6914321', '6914839', '6914909'] 2022-07-07T19:00:35.105 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/multi/{export-check', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'wsync/no}'] 2022-07-07T19:00:35.106 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{no-subvolume}', 'subvolume/{with-namespace-isolated-and-quota}', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/fs/misc}}', 'workunit/suites/ffsb}}', 'workunit/suites/fsstress}}', 'workunit/suites/fsync-tester}}', 'workunit/suites/iogen}}', 'wsync/no}'] 2022-07-07T19:00:35.107 INFO:scrape: 2022-07-07T19:00:35.109 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi104 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2022-07-07T19:00:35.110 INFO:scrape:2 jobs: ['6914489', '6914482'] 2022-07-07T19:00:35.111 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2022-07-07T19:00:35.112 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2022-07-07T19:00:35.116 INFO:scrape: 2022-07-07T19:00:35.117 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2022-07-07T19:00:35.118 INFO:scrape:8 jobs: ['6914529', '6914402', '6914450', '6914687', '6914933', '6914848', '6914604', '6914298'] 2022-07-07T19:00:35.119 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/pjd}}', 'wsync/no}'] 2022-07-07T19:00:35.120 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated-and-quota}', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/suites/pjd}}', 'wsync/no}'] 2022-07-07T19:00:35.121 INFO:scrape: 2022-07-07T19:00:35.122 INFO:scrape:Failure: "2022-07-06T19:36:14.394681+0000 mon.a (mon.0) 230 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-07-07T19:00:35.123 INFO:scrape:10 jobs: ['6914345', '6914307', '6914385', '6914917', '6914415', '6914426', '6914904', '6914309', '6914953', '6914389'] 2022-07-07T19:00:35.123 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2022-07-07T19:00:35.124 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/permission/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/ino_release_cb}', 'tasks/ior-shared-file}', 'tasks/quota}', 'tasks/recovery-fs}', 'tasks/snap-schedule}', 'tasks/strays}'] 2022-07-07T19:00:35.125 INFO:scrape: 2022-07-07T19:00:35.125 INFO:scrape:Failure: {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} 2022-07-07T19:00:35.126 INFO:scrape:17 jobs: ['6914427', '6914624', '6914394', '6914783', '6914353', '6914660', '6914712', '6914533', '6914976', '6914383', '6914453', '6914823', '6914446', '6914387', '6914346', '6914413', '6914875'] 2022-07-07T19:00:35.127 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2022-07-07T19:00:35.127 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'scrub/no', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'subvolume/{no-subvolume}', 'subvolume/{with-namespace-isolated-and-quota}', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-no-extra-options}', 'subvolume/{with-quota}', 'tasks/asok_dump_tree}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/damage}', 'tasks/workunit/dir-max-entries}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'traceless/50pc}', 'workunit/direct_io}}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}', 'workunit/suites/dbench}}', 'workunit/suites/fsx}}', 'wsync/no}', 'wsync/yes}'] 2022-07-07T19:00:35.128 INFO:scrape: 2022-07-07T19:00:35.129 INFO:scrape:Failure: Command failed on smithi099 with status 5: 'sudo systemctl stop ceph-3ad3c2b4-fda2-11ec-842d-001a4aab830c@mon.b' 2022-07-07T19:00:35.130 INFO:scrape:5 jobs: ['6914588', '6914643', '6914544', '6914864', '6914958'] 2022-07-07T19:00:35.131 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter'] 2022-07-07T19:00:35.132 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'scrub/no', 'session_timeout}', 'standby-replay', 'subvolume/{no-subvolume}', 'subvolume/{with-namespace-isolated-and-quota}', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-no-extra-options}', 'tasks/{0-check-counter', 'workunit/fs/test_o_trunc}}', 'workunit/kernel_untar_build}}', 'workunit/suites/dbench}}'] 2022-07-07T19:00:35.133 INFO:scrape: 2022-07-07T19:00:35.134 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2022-07-07T19:00:35.136 INFO:scrape:1 jobs: ['6914861'] 2022-07-07T19:00:35.137 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}'] 2022-07-07T19:00:35.138 INFO:scrape: 2022-07-07T19:00:35.139 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-07-07T19:00:35.140 INFO:scrape:5 jobs: ['6914528', '6914540', '6914725', '6914322', '6914826'] 2022-07-07T19:00:35.141 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{frag'] 2022-07-07T19:00:35.142 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'msgr-failures/osd-mds-delay', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'workunit/fs/misc}}', 'wsync/no}'] 2022-07-07T19:00:35.143 INFO:scrape: 2022-07-07T19:00:35.144 INFO:scrape:Failure: Test failure: test_query_client_ip_filter (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2022-07-07T19:00:35.144 INFO:scrape:2 jobs: ['6914810', '6914295'] 2022-07-07T19:00:35.146 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/metrics}'] 2022-07-07T19:00:35.148 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/metrics}'] 2022-07-07T19:00:35.148 INFO:scrape: 2022-07-07T19:00:35.149 INFO:scrape:Failure: Command failed on smithi103 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it' 2022-07-07T19:00:35.150 INFO:scrape:4 jobs: ['6914789', '6914716', '6914328', '6914555'] 2022-07-07T19:00:35.151 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'wsync/yes}'] 2022-07-07T19:00:35.152 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'scrub/no', 'session_timeout}', 'standby-replay', 'subvolume/{with-no-extra-options}', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/kernel_untar_build}}', 'workunit/suites/dbench}}', 'wsync/yes}'] 2022-07-07T19:00:35.153 INFO:scrape: 2022-07-07T19:00:35.154 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi017.front.sepia.ceph.com: ['type=AVC msg=audit(1657174316.866:193): avc: denied { node_bind } for pid=1407 comm="ping" saddr=172.21.15.17 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2022-07-07T19:00:35.155 INFO:scrape:2 jobs: ['6914884', '6914577'] 2022-07-07T19:00:35.156 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'objectstore-ec/bluestore-ec-root', 'osd}'] 2022-07-07T19:00:35.157 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'session_timeout', 'session_timeout}', 'tasks/dbench', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/valgrind}'] 2022-07-07T19:00:35.158 INFO:scrape: 2022-07-07T19:00:35.159 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2022-07-07T19:00:35.160 INFO:scrape:1 jobs: ['6914924'] 2022-07-07T19:00:35.161 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated}', 'tasks/{0-check-counter', 'workunit/suites/iogen}}', 'wsync/no}'] 2022-07-07T19:00:35.162 INFO:scrape: 2022-07-07T19:00:35.163 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2022-07-07T19:00:35.164 INFO:scrape:1 jobs: ['6914692'] 2022-07-07T19:00:35.165 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2022-07-07T19:00:35.166 INFO:scrape: 2022-07-07T19:00:35.167 INFO:scrape:Failure: {'smithi121.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['libgcrypt', 'systemd'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: libgcrypt-1.8.5-7.el8.x86_64: Cannot download, all mirrors were already tried without success', 'results': []}} 2022-07-07T19:00:35.167 INFO:scrape:2 jobs: ['6914386', '6914420'] 2022-07-07T19:00:35.168 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2022-07-07T19:00:35.169 INFO:scrape:suites union: ['1-ceph', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-logrotate}', '2-workunit/suites/fsstress}}', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'roles', 'session_timeout', 'syntax}', 'tasks/{0-from/v16.2.4', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-07-07T19:00:35.170 INFO:scrape: 2022-07-07T19:00:35.171 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2022-07-07T19:00:35.171 INFO:scrape:3 jobs: ['6914618', '6914401', '6914451'] 2022-07-07T19:00:35.173 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}'] 2022-07-07T19:00:35.173 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2022-07-07T19:00:35.174 INFO:scrape: 2022-07-07T19:00:35.175 INFO:scrape:Failure: "2022-07-06T20:28:47.763062+0000 mds.e (mds.0) 27 : cluster [WRN] client.4788 isn't responding to mclientcaps(revoke), ino 0x10000003fd7 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 301.965420 seconds ago" in cluster log 2022-07-07T19:00:35.175 INFO:scrape:3 jobs: ['6914333', '6914432', '6914960'] 2022-07-07T19:00:35.176 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'msgr-failures/none', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-07-07T19:00:35.177 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-07-07T19:00:35.178 INFO:scrape: 2022-07-07T19:00:35.179 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2022-07-07T19:00:35.179 INFO:scrape:1 jobs: ['6914315'] 2022-07-07T19:00:35.180 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2022-07-07T19:00:35.181 INFO:scrape: 2022-07-07T19:00:35.182 INFO:scrape:Failure: "2022-07-07T06:27:48.791777+0000 osd.5 (osd.5) 36 : cluster [ERR] 5.16s0 deep-scrub : stat mismatch, got 0/3 objects, 0/0 clones, 0/3 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/7993 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log 2022-07-07T19:00:35.183 INFO:scrape:1 jobs: ['6914850'] 2022-07-07T19:00:35.184 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-07-07T19:00:35.185 INFO:scrape: 2022-07-07T19:00:35.186 INFO:scrape:Failure: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' 2022-07-07T19:00:35.186 INFO:scrape:1 jobs: ['6914382'] 2022-07-07T19:00:35.187 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-07-07T19:00:35.188 INFO:scrape: 2022-07-07T19:00:35.189 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) 2022-07-07T19:00:35.190 INFO:scrape:1 jobs: ['6914599'] 2022-07-07T19:00:35.191 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2022-07-07T19:00:35.191 INFO:scrape: 2022-07-07T19:00:35.192 INFO:scrape:Crash: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' 2022-07-07T19:00:35.193 INFO:scrape:ceph version 17.0.0-13395-ga007ff09 (a007ff09098d94fc80260aff6116276eb25fc941) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f4721612ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f4722ec1fa2] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2c2165) [0x7f4722ec2165] 6: ceph-mds(+0x28f423) [0x55fed228f423] 7: (MDCache::dispatch_fragment_dir(boost::intrusive_ptr&)+0x4f9) [0x55fed231e1c9] 8: (MDCache::fragment_frozen(boost::intrusive_ptr&, int)+0x2d4) [0x55fed231f004] 9: (MDCache::fragment_mark_and_complete(boost::intrusive_ptr&)+0xa5c) [0x55fed233024c] 10: (MDCache::merge_dir(CInode*, frag_t)+0x5ab) [0x55fed233108b] 11: ceph-mds(+0x41b492) [0x55fed241b492] 12: (Context::complete(int)+0xd) [0x55fed21cd64d] 13: (CommonSafeTimer::timer_thread()+0x18b) [0x7f4722fd95ab] 14: (CommonSafeTimerThread::entry()+0x11) [0x7f4722fdb5b1] 15: /lib64/libpthread.so.0(+0x81cf) [0x7f47216081cf] 16: clone() 2022-07-07T19:00:35.194 INFO:scrape:1 jobs: ['6914319'] 2022-07-07T19:00:35.195 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2022-07-07T19:00:35.196 INFO:scrape: 2022-07-07T19:00:35.240 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/ failed: 57 dead: 23 running: 1 waiting: 0 queued: 0 passed: 370 Fail ================================================================= [6914295] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:23:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914295/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914295/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8c9ef33419074d9e80ea7c28c88ff989 Test failure: test_query_client_ip_filter (tasks.cephfs.test_mds_metrics.TestMDSMetrics) [6914298] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:25:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914298/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914298/ sentry: https://sentry.ceph.com/organizations/ceph/?query=861b3513de50402abd57892af8e6224a Command failed (workunit test suites/pjd.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6914307] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:59:06 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914307/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914307/ "2022-07-06T18:55:25.628828+0000 mon.a (mon.0) 222 : cluster [WRN] Health check failed: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6914309] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:17:55 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914309/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914309/ "2022-07-06T18:44:02.652363+0000 mon.a (mon.0) 116 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6914315] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:34:39 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914315/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914315/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab5aa65a2e52475f936a22989b69fe79 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [6914319] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:55:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914319/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914319/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43d111a9dda24370a6566ecbaac1e4ce Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6914321] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/misc}} ----------------------------------------------------------------- time: 00:14:39 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914321/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914321/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e88e885fea2b4e949dc5f4bb46f5b0c4 Command failed on smithi084 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:a007ff09098d94fc80260aff6116276eb25fc941 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bff7ae6e-fd5d-11ec-842d-001a4aab830c -- ceph orch daemon add osd smithi084:vg_nvme/lv_3' [6914322] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:21:10 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914322/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914322/ sentry: https://sentry.ceph.com/organizations/ceph/?query=416f3902b6a94bbfa1da8b6386fdee9c reached maximum tries (90) after waiting for 540 seconds [6914328] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:53:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914328/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914328/ Command failed on smithi017 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' [6914333] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 04:27:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914333/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914333/ "2022-07-06T20:28:47.763062+0000 mds.e (mds.0) 27 : cluster [WRN] client.4788 isn't responding to mclientcaps(revoke), ino 0x10000003fd7 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 301.965420 seconds ago" in cluster log [6914345] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:16:08 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914345/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914345/ "2022-07-06T19:36:14.394681+0000 mon.a (mon.0) 230 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6914382] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 06:21:26 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914382/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914382/ sentry: https://sentry.ceph.com/organizations/ceph/?query=80d0ac9313454eb7833ad5a68862d498 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6914385] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:55:30 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914385/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914385/ "2022-07-06T20:11:05.528991+0000 mon.a (mon.0) 390 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [6914389] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:48 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914389/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914389/ "2022-07-06T20:07:52.739823+0000 mon.a (mon.0) 127 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6914401] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:07:17 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914401/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=567bdb4c886347d48e72b63a505db23f Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [6914402] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:25:13 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914402/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88dd0d2b6cc54db68b6ac6fcd2bf27d2 Command failed (workunit test suites/pjd.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6914415] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:40:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914415/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914415/ "2022-07-06T20:48:55.573835+0000 mon.a (mon.0) 1934 : cluster [WRN] Health check failed: Degraded data redundancy: 1/2770 objects degraded (0.036%), 1 pg degraded (PG_DEGRADED)" in cluster log [6914419] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:15:04 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914419/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914419/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5591481363c7429e808aada9b6a8340e Command failed on smithi098 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:a007ff09098d94fc80260aff6116276eb25fc941 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 29975be2-fd6a-11ec-842d-001a4aab830c -- ceph orch daemon add osd smithi098:vg_nvme/lv_3' [6914426] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} ----------------------------------------------------------------- time: 00:16:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914426/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914426/ "2022-07-06T20:35:12.944869+0000 mon.a (mon.0) 128 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6914432] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 03:44:13 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914432/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914432/ "2022-07-06T21:10:32.403937+0000 mds.d (mds.2) 10 : cluster [WRN] client.4783 isn't responding to mclientcaps(revoke), ino 0x300000006e3 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.067839 seconds ago" in cluster log [6914450] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:25:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914450/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914450/ sentry: https://sentry.ceph.com/organizations/ceph/?query=69b23dafc0c74a39b7de6866d6743b10 Command failed (workunit test suites/pjd.sh) on smithi072 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6914451] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:07:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914451/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914451/ sentry: https://sentry.ceph.com/organizations/ceph/?query=facf4c19b9f943a9a7d5cbc4663c0af7 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [6914482] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:45:37 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914482/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914482/ sentry: https://sentry.ceph.com/organizations/ceph/?query=270f42711f984d8b901a8405dd99d890 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi019 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6914489] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:32:07 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914489/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914489/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f660e319987c4dffa1d14bd3c3cdf925 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi104 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [6914495] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:14:30 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914495/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914495/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dae23148639143608d2561fb8f8ff6bd Command failed on smithi146 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:a007ff09098d94fc80260aff6116276eb25fc941 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea38b262-fd9a-11ec-842d-001a4aab830c -- ceph orch daemon add osd smithi146:vg_nvme/lv_4' [6914528] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:25:03 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914528/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914528/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ca05ea8bcfc4b2fa0cc90a37df57cdc reached maximum tries (90) after waiting for 540 seconds [6914529] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:24:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914529/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914529/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2dc472bed544d95950723c4d1942ced Command failed (workunit test suites/pjd.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6914540] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/misc}} ----------------------------------------------------------------- time: 01:08:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914540/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914540/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6914544] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:17:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914544/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914544/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce1ca621cd3747a785f9b95ccb7f90a3 Command failed on smithi075 with status 5: 'sudo systemctl stop ceph- fceb2724-fd9e-11ec-842d-001a4aab830c@mon.b' [6914555] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:56:52 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914555/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914555/ Command failed on smithi035 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' [6914577] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:18:30 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914577/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914577/ SELinux denials found on ubuntu@smithi059.front.sepia.ceph.com: ['type=AVC msg=audit(1657162139.804:203): avc: denied { node_bind } for pid=1433 comm="ping" saddr=172.21.15.59 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6914588] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:20:07 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914588/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914588/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4ecd8f80f774c50b48e1986fcff36c2 Command failed on smithi099 with status 5: 'sudo systemctl stop ceph-3ad3c2b4-fda2-11ec-842d-001a4aab830c@mon.b' [6914599] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:16:12 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914599/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914599/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fedeca1fedb45008ee6986e9e04615b Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) [6914604] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:25:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914604/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914604/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b526dc42143494cbd0d432665d1e1c5 Command failed (workunit test suites/pjd.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6914618] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:49:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914618/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914618/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6220e8da9e14240a9e1b367186af126 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [6914643] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:18:12 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914643/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914643/ sentry: https://sentry.ceph.com/organizations/ceph/?query=351aae945c1a4fb3be0643240b9e575b Command failed on smithi159 with status 5: 'sudo systemctl stop ceph-7321e020-fda6-11ec-842d-001a4aab830c@mon.b' [6914687] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:26:35 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914687/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914687/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f7b245b56e64215a12aff534241debc Command failed (workunit test suites/pjd.sh) on smithi088 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6914692] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 03:40:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914692/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914692/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c9914ad687d4cdebb836f3caa5c9238 Command failed (workunit test suites/fsstress.sh) on smithi133 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [6914716] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:58:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914716/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914716/ Command failed on smithi083 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' [6914725] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:25:06 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914725/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=caed80e15fac409ea4934d032f4a032c reached maximum tries (90) after waiting for 540 seconds [6914789] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:58:12 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914789/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914789/ Command failed on smithi103 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it' [6914810] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:34:49 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914810/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5289b972b44441a878cd06ec770fa9a Test failure: test_query_client_ip_filter (tasks.cephfs.test_mds_metrics.TestMDSMetrics) [6914826] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:22:58 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914826/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914826/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c18db64e3e924770993dfb3a25d3fb57 reached maximum tries (90) after waiting for 540 seconds [6914839] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:14:34 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914839/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914839/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd079520a80046a0a4942d487c6b8550 Command failed on smithi172 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:a007ff09098d94fc80260aff6116276eb25fc941 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 34bd0f8e-fdbb-11ec-842d-001a4aab830c -- ceph orch daemon add osd smithi172:vg_nvme/lv_4' [6914848] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:25:39 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914848/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914848/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35ecea90042542158af68f9793b87fc8 Command failed (workunit test suites/pjd.sh) on smithi103 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6914850] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:33:07 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914850/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914850/ "2022-07-07T06:27:48.791777+0000 osd.5 (osd.5) 36 : cluster [ERR] 5.16s0 deep-scrub : stat mismatch, got 0/3 objects, 0/0 clones, 0/3 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/7993 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log [6914861] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:20:56 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914861/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914861/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae9318455b904dbf87d2890de35ff703 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [6914864] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:19:30 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914864/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd2cdea1396c4ad8ad7712058b9b49f2 Command failed on smithi109 with status 5: 'sudo systemctl stop ceph-a5453e14-fdbd-11ec-842d-001a4aab830c@mon.b' [6914884] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 01:02:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914884/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914884/ SELinux denials found on ubuntu@smithi017.front.sepia.ceph.com: ['type=AVC msg=audit(1657174316.866:193): avc: denied { node_bind } for pid=1407 comm="ping" saddr=172.21.15.17 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6914904] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:15:12 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914904/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914904/ "2022-07-07T06:34:51.328926+0000 mon.a (mon.0) 117 : cluster [WRN] Health check failed: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6914909] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:14:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914909/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914909/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d10b52571d84f64ac6cedbe4bea7383 Command failed on smithi082 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:a007ff09098d94fc80260aff6116276eb25fc941 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f9957d98-fdbe-11ec-842d-001a4aab830c -- ceph orch daemon add osd smithi082:vg_nvme/lv_4' [6914917] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:24:04 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914917/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914917/ "2022-07-07T06:49:04.097047+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6914924] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/iogen}} ----------------------------------------------------------------- time: 01:08:09 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914924/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914924/ The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} [6914933] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:24:49 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914933/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914933/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec18b895f492427990de15fdd996116d Command failed (workunit test suites/pjd.sh) on smithi083 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a007ff09098d94fc80260aff6116276eb25fc941 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [6914953] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:52:58 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914953/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914953/ "2022-07-07T07:00:38.192323+0000 mon.a (mon.0) 123 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6914958] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:20:46 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914958/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914958/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5a2678c5a344a48baa3789fe6fc2b1b Command failed on smithi035 with status 5: 'sudo systemctl stop ceph-5c99174e-fd09-11ec-842d-001a4aab830c@mon.b' [6914960] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 03:40:33 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914960/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914960/ "2022-07-07T07:23:09.362271+0000 mds.c (mds.0) 4 : cluster [WRN] client.4768 isn't responding to mclientcaps(revoke), ino 0x10000004bdc pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.069757 seconds ago" in cluster log Dead ================================================================= [6914325] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914325/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914325/ hit max job timeout [6914346] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:11:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914346/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914346/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db858938cf8f44429602531f3e2e5d7e {'smithi132.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914353] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} ----------------------------------------------------------------- time: 00:11:13 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914353/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914353/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33794bb0b6af4cd7bd36d034b82aa51b {'smithi169.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914383] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:11:27 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914383/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=225a52dada9049d6904f9522ff9abb0a {'smithi132.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914386] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:10:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914386/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914386/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0be0bdf63a354d73bea2efdcd8342b41 {'smithi121.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['libgcrypt', 'systemd'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: libgcrypt-1.8.5-7.el8.x86_64: Cannot download, all mirrors were already tried without success', 'results': []}} [6914387] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:11:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914387/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914387/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a651610e34df4acd8976e814366d9cbc {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914394] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:11:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914394/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914394/ sentry: https://sentry.ceph.com/organizations/ceph/?query=991583c425414500a1611dda7a0888a1 {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914413] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:07:33 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914413/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6246e98311e420889a05d648324cc58 {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914420] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:10:37 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914420/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914420/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b4c52fefb2a5432fb38be2e60308a71d {'smithi053.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['libgcrypt', 'systemd'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'latest', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: libgcrypt-1.8.5-7.el8.x86_64: Cannot download, all mirrors were already tried without success', 'results': []}} [6914427] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:11:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914427/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914427/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f561796f0b7c4a11be58bfcaf4baff3b {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914446] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:12:04 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914446/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914446/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e133974dce4d4f80af7225727cdc0ef3 {'smithi090.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914453] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/direct_io}} ----------------------------------------------------------------- time: 00:12:08 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914453/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914453/ sentry: https://sentry.ceph.com/organizations/ceph/?query=caa5d6479bbb475ba0935bcbdae39608 {'smithi143.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914533] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:11:13 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914533/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914533/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68fd8bb055024a358e34a239857c1324 {'smithi169.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914551] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914551/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914551/ hit max job timeout [6914624] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/direct_io}} ----------------------------------------------------------------- time: 00:11:26 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914624/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914624/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8c84bea80b7f434eb40d6c56afe844e7 {'smithi143.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914640] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914640/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914640/ hit max job timeout [6914660] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:11:57 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914660/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914660/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7724d7618274580b28118d038ef4939 {'smithi143.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914712] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:12:02 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914712/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914712/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1378c95556e3454980c0df1b02d82d67 {'smithi132.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914783] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/misc}} ----------------------------------------------------------------- time: 00:07:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914783/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c6864fdaa75434b976482b4f28c49b0 {'smithi050.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914794] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914794/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914794/ hit max job timeout [6914823] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:11:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914823/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914823/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b75444f8853249ddb32b5aae4244ade3 {'smithi039.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914875] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:12:54 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914875/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914875/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61282a4c04a7472e8c8aa2725a88f613 {'smithi090.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} [6914976] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:11:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914976/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914976/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f43c709b683a48ca920bff528ea7cdf5 {'smithi132.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} Running ================================================================= [6914955] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/blogbench}} info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914955/ Pass ================================================================= [6914294] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:17:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914294/ [6914296] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/iozone}} time: 00:34:34 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914296/ [6914297] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:17:18 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914297/ [6914299] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:19:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914299/ [6914300] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:18 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914300/ [6914301] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:17:10 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914301/ [6914302] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:52:41 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914302/ [6914303] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:21:32 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914303/ [6914304] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:15:24 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914304/ [6914305] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror} time: 02:14:16 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914305/ [6914306] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:42:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914306/ [6914308] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:57:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914308/ [6914310] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:45:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914310/ [6914311] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:20:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914311/ [6914312] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:17 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914312/ [6914313] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop} time: 00:16:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914313/ [6914314] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:39:26 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914314/ [6914316] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:21:45 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914316/ [6914317] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:25:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914317/ [6914318] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/direct_io}} time: 00:29:33 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914318/ [6914320] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:21:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914320/ [6914323] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/kernel_untar_build}} time: 03:18:07 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914323/ [6914324] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:36:36 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914324/ [6914326] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:19:01 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914326/ [6914327] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:18:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914327/ [6914329] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:10 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914329/ [6914330] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:21:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914330/ [6914331] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/ffsb}} time: 00:32:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914331/ [6914332] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:41:36 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914332/ [6914334] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/norstats}} time: 00:36:29 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914334/ [6914335] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:44:16 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914335/ [6914336] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsstress}} time: 00:26:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914336/ [6914337] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:12:33 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914337/ [6914338] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:21:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914338/ [6914339] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsx}} time: 00:41:32 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914339/ [6914340] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:18:39 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914340/ [6914341] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsync-tester}} time: 00:24:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914341/ [6914342] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:16:09 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914342/ [6914343] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:20:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914343/ [6914344] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/fs/test_o_trunc}} time: 00:40:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914344/ [6914347] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/iogen}} time: 01:06:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914347/ [6914348] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:16:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914348/ [6914349] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:26:59 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914349/ [6914350] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/iozone}} time: 00:31:06 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914350/ [6914351] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:26:14 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914351/ [6914352] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:34:52 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914352/ [6914354] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/pjd}} time: 00:24:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914354/ [6914355] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:34 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914355/ [6914356] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:16:49 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914356/ [6914357] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:17:45 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914357/ [6914358] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/direct_io}} time: 00:38:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914358/ [6914359] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:19:46 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914359/ [6914360] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/misc}} time: 00:51:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914360/ [6914361] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:24:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914361/ [6914362] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/kernel_untar_build}} time: 00:58:01 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914362/ [6914363] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:20:21 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914363/ [6914365] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:36:21 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914365/ [6914367] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:02 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914367/ [6914369] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/blogbench}} time: 00:39:39 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914369/ [6914370] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:51:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914370/ [6914371] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:24 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914371/ [6914372] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:48 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914372/ [6914373] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/dbench}} time: 01:17:49 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914373/ [6914374] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:26:21 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914374/ [6914375] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:19:21 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914375/ [6914376] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/ffsb}} time: 00:52:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914376/ [6914377] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:18:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914377/ [6914378] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914378/ [6914379] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/fs/norstats}} time: 00:30:48 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914379/ [6914380] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:18:32 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914380/ [6914381] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsstress}} time: 00:26:13 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914381/ [6914384] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:22:30 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914384/ [6914388] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsync-tester}} time: 00:25:08 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914388/ [6914390] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:23:49 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914390/ [6914391] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:06:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914391/ [6914392] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} time: 00:57:29 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914392/ [6914393] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:30:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914393/ [6914395] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/test_o_trunc}} time: 00:38:50 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914395/ [6914396] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:16:27 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914396/ [6914397] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/iogen}} time: 00:35:07 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914397/ [6914398] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:12:29 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914398/ [6914399] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:18:37 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914399/ [6914400] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/iozone}} time: 00:35:50 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914400/ [6914403] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:47:34 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914403/ [6914404] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:21:07 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914404/ [6914405] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:30:27 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914405/ [6914406] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/direct_io}} time: 00:30:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914406/ [6914407] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:26:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914407/ [6914408] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:18:52 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914408/ [6914409] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/fs/misc}} time: 00:46:48 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914409/ [6914410] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:20:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914410/ [6914411] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/kernel_untar_build}} time: 02:58:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914411/ [6914412] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:22:06 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914412/ [6914414] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:23:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914414/ [6914416] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:36:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914416/ [6914417] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/dbench}} time: 02:18:14 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914417/ [6914418] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:23:03 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914418/ [6914421] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:42:46 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914421/ [6914422] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/norstats}} time: 00:36:52 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914422/ [6914423] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:19:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914423/ [6914424] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:18:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914424/ [6914425] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:19 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914425/ [6914428] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsstress}} time: 00:27:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914428/ [6914429] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:26 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914429/ [6914430] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsx}} time: 00:41:14 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914430/ [6914431] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:22:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914431/ [6914433] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsync-tester}} time: 00:24:30 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914433/ [6914434] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:17:14 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914434/ [6914435] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:37:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914435/ [6914436] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/fs/test_o_trunc}} time: 00:41:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914436/ [6914437] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 02:04:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914437/ [6914438] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:49:57 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914438/ [6914439] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:21:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914439/ [6914440] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/iogen}} time: 01:05:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914440/ [6914441] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:39:52 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914441/ [6914442] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:19:25 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914442/ [6914443] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/iozone}} time: 00:32:33 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914443/ [6914444] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:17:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914444/ [6914448] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:21:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914448/ [6914455] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:25:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914455/ [6914457] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/misc}} time: 00:52:12 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914457/ [6914459] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:23:24 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914459/ [6914460] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:15:17 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914460/ [6914462] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/kernel_untar_build}} time: 01:01:04 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914462/ [6914464] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:17:55 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914464/ [6914466] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:45:34 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914466/ [6914468] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/blogbench}} time: 00:48:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914468/ [6914469] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:18:58 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914469/ [6914471] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:23:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914471/ [6914473] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/dbench}} time: 01:24:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914473/ [6914475] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:21:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914475/ [6914477] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:22:48 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914477/ [6914478] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:54:35 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914478/ [6914480] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:51:34 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914480/ [6914484] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:23:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914484/ [6914486] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:37:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914486/ [6914487] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/ffsb}} time: 00:55:41 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914487/ [6914491] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/norstats}} time: 00:31:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914491/ [6914493] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:08:52 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914493/ [6914497] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:20:38 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914497/ [6914498] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:21:19 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914498/ [6914500] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:17:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914500/ [6914502] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsx}} time: 00:47:56 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914502/ [6914504] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914504/ [6914506] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:21:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914506/ [6914508] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsync-tester}} time: 00:24:32 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914508/ [6914509] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:49 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914509/ [6914511] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/test_o_trunc}} time: 00:41:00 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914511/ [6914513] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:48:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914513/ [6914515] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:14 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914515/ [6914517] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914517/ [6914519] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:13:25 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914519/ [6914520] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/iogen}} time: 00:35:37 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914520/ [6914522] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:08:35 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914522/ [6914524] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/iozone}} time: 00:36:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914524/ [6914526] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:36 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914526/ [6914531] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:21:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914531/ [6914535] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:42:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914535/ [6914537] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/direct_io}} time: 00:29:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914537/ [6914539] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:17:25 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914539/ [6914542] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:52:36 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914542/ [6914546] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:14:35 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914546/ [6914548] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:15:57 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914548/ [6914549] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:16:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914549/ [6914553] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914553/ [6914557] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:23:27 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914557/ [6914558] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/ffsb}} time: 00:34:07 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914558/ [6914560] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:16:04 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914560/ [6914562] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:32:27 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914562/ [6914564] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:12:10 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914564/ [6914566] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/norstats}} time: 00:37:18 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914566/ [6914568] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:24:16 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914568/ [6914569] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsstress}} time: 00:35:24 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914569/ [6914571] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:05:37 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914571/ [6914573] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:40:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914573/ [6914575] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:08:07 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914575/ [6914578] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsx}} time: 00:43:10 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914578/ [6914580] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:21:49 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914580/ [6914582] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:15:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914582/ [6914584] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsync-tester}} time: 00:24:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914584/ [6914586] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:58 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914586/ [6914589] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:21:58 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914589/ [6914591] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/iogen}} time: 01:08:03 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914591/ [6914593] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:18:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914593/ [6914595] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:42:00 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914595/ [6914597] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:46:44 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914597/ [6914600] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/iozone}} time: 00:31:57 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914600/ [6914602] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:04:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914602/ [6914606] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:19:01 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914606/ [6914607] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914607/ [6914609] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:25:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914609/ [6914611] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:19:30 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914611/ [6914613] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:59:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914613/ [6914615] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:35:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914615/ [6914616] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914616/ [6914620] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:59:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914620/ [6914622] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:07:13 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914622/ [6914625] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:19:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914625/ [6914627] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/fs/misc}} time: 01:02:01 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914627/ [6914629] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:13:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914629/ [6914631] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:21:17 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914631/ [6914633] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/kernel_untar_build}} time: 00:58:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914633/ [6914635] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:16:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914635/ [6914636] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:37:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914636/ [6914638] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:01:01 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914638/ [6914642] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:17:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914642/ [6914645] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:39:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914645/ [6914647] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:17:49 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914647/ [6914649] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/ffsb}} time: 00:58:12 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914649/ [6914651] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:19:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914651/ [6914653] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/norstats}} time: 00:33:14 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914653/ [6914654] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:24:36 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914654/ [6914656] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:17:33 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914656/ [6914658] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsstress}} time: 00:27:21 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914658/ [6914662] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914662/ [6914663] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/fsx}} time: 00:51:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914663/ [6914665] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:46 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914665/ [6914667] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:04:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914667/ [6914669] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsync-tester}} time: 00:25:21 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914669/ [6914671] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:47:17 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914671/ [6914673] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/test_o_trunc}} time: 00:41:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914673/ [6914674] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:28:16 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914674/ [6914676] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:24:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914676/ [6914678] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/iogen}} time: 00:36:35 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914678/ [6914680] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:17:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914680/ [6914682] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:20:49 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914682/ [6914683] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/iozone}} time: 00:37:04 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914683/ [6914685] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914685/ [6914689] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:51:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914689/ [6914691] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:00:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914691/ [6914694] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/direct_io}} time: 00:31:01 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914694/ [6914696] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:14:25 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914696/ [6914698] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/misc}} time: 00:47:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914698/ [6914700] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:16 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914700/ [6914701] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:19:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914701/ [6914703] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/kernel_untar_build}} time: 02:54:19 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914703/ [6914705] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:21:55 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914705/ [6914707] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:21:41 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914707/ [6914709] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/blogbench}} time: 00:43:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914709/ [6914710] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:25:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914710/ [6914714] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:17:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914714/ [6914718] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:53:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914718/ [6914719] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/ffsb}} time: 00:33:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914719/ [6914721] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:26:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914721/ [6914723] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:21:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914723/ [6914727] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:18:35 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914727/ [6914728] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:52 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914728/ [6914730] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/norstats}} time: 00:35:10 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914730/ [6914732] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:16:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914732/ [6914734] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:16:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914734/ [6914736] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/fsstress}} time: 00:26:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914736/ [6914738] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:21:44 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914738/ [6914739] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/fsx}} time: 00:41:03 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914739/ [6914741] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:21:34 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914741/ [6914743] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:16:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914743/ [6914745] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsync-tester}} time: 00:24:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914745/ [6914747] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:15:41 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914747/ [6914749] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:54:31 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914749/ [6914750] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:44:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914750/ [6914752] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:18:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914752/ [6914754] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:22:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914754/ [6914756] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:17:37 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914756/ [6914758] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/test_o_trunc}} time: 00:41:21 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914758/ [6914759] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:14:35 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914759/ [6914761] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:15:46 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914761/ [6914763] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/iogen}} time: 01:06:02 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914763/ [6914765] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:41:08 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914765/ [6914767] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:17:09 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914767/ [6914768] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:24:07 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914768/ [6914770] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/iozone}} time: 00:31:46 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914770/ [6914772] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:29 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914772/ [6914774] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/pjd}} time: 00:24:36 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914774/ [6914776] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:20:52 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914776/ [6914777] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/direct_io}} time: 00:43:57 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914777/ [6914779] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914779/ [6914781] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:22:29 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914781/ [6914785] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:01:33 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914785/ [6914787] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:31:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914787/ [6914790] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:15:56 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914790/ [6914792] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:33:06 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914792/ [6914796] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:19:56 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914796/ [6914798] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:14:58 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914798/ [6914799] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/dbench}} time: 01:23:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914799/ [6914801] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:24:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914801/ [6914803] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/ffsb}} time: 00:56:17 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914803/ [6914805] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:17:19 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914805/ [6914807] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:02:36 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914807/ [6914808] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/norstats}} time: 00:36:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914808/ [6914812] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:21:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914812/ [6914814] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:54 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914814/ [6914816] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:18:15 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914816/ [6914817] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:53 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914817/ [6914819] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/fsstress}} time: 00:26:35 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914819/ [6914821] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:16:45 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914821/ [6914825] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:42:29 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914825/ [6914828] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/fsync-tester}} time: 00:24:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914828/ [6914830] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:24:44 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914830/ [6914832] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:19:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914832/ [6914834] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/test_o_trunc}} time: 00:39:17 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914834/ [6914835] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:16:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914835/ [6914837] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:53:22 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914837/ [6914841] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:39:18 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914841/ [6914843] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/iozone}} time: 00:35:45 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914843/ [6914845] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:50:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914845/ [6914846] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:20:29 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914846/ [6914852] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:38:21 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914852/ [6914854] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/direct_io}} time: 00:31:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914854/ [6914855] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:18:00 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914855/ [6914857] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:25:30 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914857/ [6914859] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/misc}} time: 00:52:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914859/ [6914863] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:19:32 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914863/ [6914866] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:43:45 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914866/ [6914868] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:28:27 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914868/ [6914870] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:29:24 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914870/ [6914872] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/blogbench}} time: 00:38:02 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914872/ [6914873] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:22:51 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914873/ [6914877] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:19:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914877/ [6914879] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:24:03 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914879/ [6914881] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:29:11 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914881/ [6914882] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:27:46 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914882/ [6914886] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:39:41 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914886/ [6914888] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:15:00 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914888/ [6914890] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/ffsb}} time: 00:34:12 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914890/ [6914891] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:19:59 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914891/ [6914893] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:43:02 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914893/ [6914895] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:24:01 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914895/ [6914897] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/norstats}} time: 00:38:01 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914897/ [6914899] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:18:14 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914899/ [6914900] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsstress}} time: 00:34:46 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914900/ [6914902] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:28:39 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914902/ [6914906] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsx}} time: 00:40:21 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914906/ [6914908] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:56:09 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914908/ [6914911] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:18:29 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914911/ [6914913] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914913/ [6914915] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/test_o_trunc}} time: 00:42:02 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914915/ [6914918] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:18:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914918/ [6914920] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:23:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914920/ [6914922] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:26:47 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914922/ [6914926] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:56 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914926/ [6914928] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:17:50 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914928/ [6914929] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/iozone}} time: 00:32:43 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914929/ [6914931] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:16:48 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914931/ [6914935] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:32:34 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914935/ [6914937] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:16:17 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914937/ [6914939] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/direct_io}} time: 00:38:28 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914939/ [6914940] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:13:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914940/ [6914942] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:24:13 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914942/ [6914944] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/misc}} time: 00:53:20 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914944/ [6914946] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914946/ [6914947] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/kernel_untar_build}} time: 00:59:52 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914947/ [6914949] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:24:04 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914949/ [6914951] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:43:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914951/ [6914957] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:16:55 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914957/ [6914962] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:49:44 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914962/ [6914964] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/ffsb}} time: 00:57:40 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914964/ [6914965] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:18:42 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914965/ [6914967] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:18:39 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914967/ [6914969] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/norstats}} time: 00:32:44 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914969/ [6914971] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:50 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914971/ [6914973] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/fsstress}} time: 00:26:09 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914973/ [6914974] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:25:36 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914974/ [6914978] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:18:01 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914978/ [6914980] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:35:27 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914980/ [6914982] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/fsync-tester}} time: 00:25:14 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914982/ [6914984] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:22:56 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914984/ [6914985] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:20:10 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914985/ [6914987] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:05:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914987/ [6914989] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/test_o_trunc}} time: 00:39:23 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914989/ [6914991] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:20:05 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914991/ [6914992] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/iogen}} time: 00:35:39 info: http://pulpito.ceph.com/vshankar-2022-07-05_17:37:39-fs-wip-vshankar-testing1-20220705-144105-testing-default-smithi/6914992/