2023-04-01T06:55:03.908 INFO:root:teuthology version: 0.0.1.dev91+g5338cb9 2023-04-01T06:55:03.933 INFO:teuthology.results:subset: '111/12000' 2023-04-01T06:55:03.933 INFO:teuthology.results:seed: '7501' 2023-04-01T09:28:12.559 INFO:root:teuthology version: 0.0.1.dev91+g5338cb9 2023-04-01T09:28:12.584 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-01T20:55:12.342 INFO:teuthology.results:Tests finished! gathering results... 2023-04-01T20:55:30.376 INFO:scrape:Found 188 jobs 2023-04-01T20:55:34.201 WARNING:scrape:Misunderstood line: 2023-04-01T20:55:46.176 INFO:scrape:Found 12 distinct failure reasons 2023-04-01T20:55:46.177 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-04-01T20:55:46.177 INFO:scrape:2 jobs: ['7229524', '7229362'] 2023-04-01T20:55:46.178 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '3-snaps/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'n/5', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}'] 2023-04-01T20:55:46.178 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-04-01T20:55:46.179 INFO:scrape: 2023-04-01T20:55:46.179 INFO:scrape:Failure: HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) 2023-04-01T20:55:46.179 INFO:scrape:31 jobs 2023-04-01T20:55:46.180 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-04-01T20:55:46.180 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/ffsb}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'bluestore-bitmap', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1-mds-1-client', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'session_timeout', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cephfs_misc_tests}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client}', 'tasks/damage}', 'tasks/forward-scrub}', 'tasks/fscrypt-common}', 'tasks/fscrypt-ffsb}', 'tasks/fscrypt-iozone}', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/mirror}', 'tasks/openfiletable}', 'tasks/snap-schedule}', 'tasks/snap_schedule_snapdir}', 'tasks/strays}', 'tasks/truncate_delay}', 'tasks/volumes/{overrides', 'tasks/workunit/quota}', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/misc}}', 'thrashosds-health}', 'traceless/50pc}'] 2023-04-01T20:55:46.181 INFO:scrape: 2023-04-01T20:55:46.181 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-04-01T20:55:46.181 INFO:scrape:1 jobs: ['7229465'] 2023-04-01T20:55:46.182 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2023-04-01T20:55:46.182 INFO:scrape: 2023-04-01T20:55:46.182 INFO:scrape:Failure: Command failed on smithi081 with status 5: 'rm -f /tmp/linux-image.deb && echo linux-image-6.3.0-rc3-g6c26fb6d76aa_6.3.0-rc3-g6c26fb6d76aa-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/6c26fb6d76aa35f265ac42f88c0a40a9c564d650/ubuntu/focal/flavors/default/pool/main/l/linux-6.3.0-rc3-g6c26fb6d76aa/ --input-file=-' 2023-04-01T20:55:46.183 INFO:scrape:8 jobs: ['7229476', '7229512', '7229517', '7229451', '7229445', '7229559', '7229381', '7229479'] 2023-04-01T20:55:46.183 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}'] 2023-04-01T20:55:46.183 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/suites/ffsb}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mon-debug}', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides}}', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'session_timeout', 'session_timeout}', 'tasks/admin}', 'tasks/failover}', 'tasks/fsstress', 'tasks/multifs-auth}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}'] 2023-04-01T20:55:46.184 INFO:scrape: 2023-04-01T20:55:46.184 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-04-01T20:55:46.184 INFO:scrape:1 jobs: ['7229510'] 2023-04-01T20:55:46.185 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-04-01T20:55:46.185 INFO:scrape: 2023-04-01T20:55:46.186 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi088 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-04-01T20:55:46.186 INFO:scrape:1 jobs: ['7229506'] 2023-04-01T20:55:46.186 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-04-01T20:55:46.187 INFO:scrape: 2023-04-01T20:55:46.187 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-04-01T20:55:46.187 INFO:scrape:1 jobs: ['7229480'] 2023-04-01T20:55:46.188 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-04-01T20:55:46.188 INFO:scrape: 2023-04-01T20:55:46.188 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi145 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-04-01T20:55:46.188 INFO:scrape:1 jobs: ['7229366'] 2023-04-01T20:55:46.189 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-04-01T20:55:46.189 INFO:scrape: 2023-04-01T20:55:46.189 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) 2023-04-01T20:55:46.190 INFO:scrape:1 jobs: ['7229410'] 2023-04-01T20:55:46.190 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2023-04-01T20:55:46.190 INFO:scrape: 2023-04-01T20:55:46.191 INFO:scrape:Dead: b'2023-04-01T19:49:39.248 DEBUG:teuthology.exit:Finished running handlers' 2023-04-01T20:55:46.191 INFO:scrape:2 jobs: ['7229425', '7229501'] 2023-04-01T20:55:46.191 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'wsync/no}'] 2023-04-01T20:55:46.192 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/blogbench}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-04-01T20:55:46.192 INFO:scrape: 2023-04-01T20:55:46.192 INFO:scrape:Failure: Command failed on smithi111 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f3dbdaa-d060-11ed-9afe-001a4aab830c -- ceph orch daemon add osd smithi111:vg_nvme/lv_4' 2023-04-01T20:55:46.193 INFO:scrape:1 jobs: ['7229385'] 2023-04-01T20:55:46.193 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-04-01T20:55:46.193 INFO:scrape: 2023-04-01T20:55:46.194 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2023-04-01T20:55:46.194 INFO:scrape:2 jobs: ['7229401', '7229455'] 2023-04-01T20:55:46.194 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2023-04-01T20:55:46.195 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/fs/norstats}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-04-01T20:55:46.195 INFO:scrape: 2023-04-01T20:55:46.207 INFO:teuthology.results:Sending results to yweinste@radhat.com: Test Run: yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/ failed: 50 dead: 2 running: 0 waiting: 0 queued: 0 passed: 136 Fail ================================================================= [7229362] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:38:24 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229362/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229362/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6181436ff8244aa4802592a61a6a93b5 Command failed (workunit test suites/pjd.sh) on smithi046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7229366] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:17:26 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229366/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229366/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6aefc18bc9f2418ab4197c202791e21d Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi145 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7229368] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:08:13 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229368/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229368/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4002088fd9e24c6abee3322fac2d7845 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229380] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:07:47 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229380/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db018da3ef7d49c5bae985719c302093 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229381] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229381/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f855e34766f945df95cdddcd2d0f92b3 Command failed on smithi093 with status 5: 'rm -f /tmp/linux-image.deb && echo linux- image-6.3.0-rc3-g6c26fb6d76aa_6.3.0-rc3-g6c26fb6d76aa-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testi ng/6c26fb6d76aa35f265ac42f88c0a40a9c564d650/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-rc3-g6c26fb6d76aa/ --input-file=-' [7229385] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:17:09 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229385/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229385/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63060e7c2d4f4db38cb10cebf9f447c4 Command failed on smithi111 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3f3dbdaa-d060-11ed-9afe-001a4aab830c -- ceph orch daemon add osd smithi111:vg_nvme/lv_4' [7229387] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:07:45 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229387/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229387/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c075adb1efc46259e49aed7c97b1757 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229388] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:07:42 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229388/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfcaa212f4fc4a12b1c95cc78394af75 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229401] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:55:54 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229401/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229401/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7229402] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:08:11 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229402/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c7f6b1edf9245d2a821e27a2da5029f HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229404] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:08:08 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229404/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ff1338d3b1a4bf1a8ac33f70bdb0851 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229408] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:08:07 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229408/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0ee07f92bb54d30b640320aa834435f HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229410] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:14:09 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229410/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229410/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9897d89c1cc143658eec13d4a48daa57 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) [7229411] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} ----------------------------------------------------------------- time: 00:07:38 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229411/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229411/ sentry: https://sentry.ceph.com/organizations/ceph/?query=462981fbf04c4e2a894265fef75c61ba HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229412] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} ----------------------------------------------------------------- time: 00:07:34 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229412/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229412/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95ab196ea4334290b9b9f4809baf3964 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229415] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:07:52 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229415/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229415/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5b291b91f9b47309758068dcdcc4036 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229424] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:07:39 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229424/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229424/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87e16172b92d46d782190ea296096569 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229431] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:07:51 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229431/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229431/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d7b6509d3f114e409309c394bc182b18 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229442] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:07:37 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229442/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229442/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a15daace50a646c485cbbf58d19aa960 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229445] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229445/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229445/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2efbfb26f7d48e5a2a8c72db1cef4a1 Command failed on smithi039 with status 5: 'rm -f /tmp/linux-image.deb && echo linux- image-6.3.0-rc3-g6c26fb6d76aa_6.3.0-rc3-g6c26fb6d76aa-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testi ng/6c26fb6d76aa35f265ac42f88c0a40a9c564d650/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-rc3-g6c26fb6d76aa/ --input-file=-' [7229447] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:07:45 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229447/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229447/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0b8f516676743deac230fe3f001c262 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229451] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229451/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229451/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f631c707e24e445c9f208e82be5e5498 Command failed on smithi007 with status 5: 'rm -f /tmp/linux-image.deb && echo linux- image-6.3.0-rc3-g6c26fb6d76aa_6.3.0-rc3-g6c26fb6d76aa-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testi ng/6c26fb6d76aa35f265ac42f88c0a40a9c564d650/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-rc3-g6c26fb6d76aa/ --input-file=-' [7229455] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:29:34 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229455/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229455/ error during scrub thrashing: rank damage found: {'backtrace'} [7229457] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:07:41 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229457/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229457/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d8ea821d72c4bdb91b0e2ff6c102c92 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229462] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:07:37 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229462/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f501cdd738bc4d8b8d17fa29857f729e HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229464] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:07:35 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229464/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229464/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5287f88b87e9423fb2abb665a938152d HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229465] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:28:47 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229465/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a40759a82852461ebee0a387c69ad76f Command failed (workunit test suites/fsstress.sh) on smithi002 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7229474] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:08:09 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229474/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229474/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19986c2e65be43f3a8a1ea4536ba654f HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229475] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:07:47 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229475/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229475/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13bb113cca78438b919dd2325888d009 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229476] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229476/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229476/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c79290c7819c41c4bedc1bcbe8f312f3 Command failed on smithi081 with status 5: 'rm -f /tmp/linux-image.deb && echo linux- image-6.3.0-rc3-g6c26fb6d76aa_6.3.0-rc3-g6c26fb6d76aa-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testi ng/6c26fb6d76aa35f265ac42f88c0a40a9c564d650/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-rc3-g6c26fb6d76aa/ --input-file=-' [7229477] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:08:05 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229477/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229477/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8dddaeeb9484494b8d67d0edd965712d HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229479] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229479/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229479/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5a9cc30846e24ea7aa5d17641d2535a7 Command failed on smithi130 with status 5: 'rm -f /tmp/linux-image.deb && echo linux- image-6.3.0-rc3-g6c26fb6d76aa_6.3.0-rc3-g6c26fb6d76aa-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testi ng/6c26fb6d76aa35f265ac42f88c0a40a9c564d650/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-rc3-g6c26fb6d76aa/ --input-file=-' [7229480] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 01:57:51 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229480/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229480/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d86d42c15d6f4b348455fdeb7ba0f274 saw valgrind issues [7229486] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:07:33 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229486/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229486/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46f431ec5756452da53e729938dfa255 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229487] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 00:07:35 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229487/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229487/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa8589a6f4854bb699c1ab6e503e2007 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229491] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} ----------------------------------------------------------------- time: 00:07:57 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229491/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229491/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af877d447f694273888b22dfe331e0a3 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229500] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:08:10 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229500/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d9fc3f0dd384fff9a09abdb531df105 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229505] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:07:44 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229505/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229505/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcedb6ab9b9640ea86c773cf04cbd25c HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229506] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:13:34 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229506/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ebe0b5ac68046d8bb72917e6ea18961 Command failed (workunit test libcephfs/test.sh) on smithi088 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7229507] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:07:48 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229507/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229507/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2399aa43c744e6da86f072deb5e3987 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229510] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:29:57 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229510/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229510/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01f2e34e69744a3891f7279287c3aa9c Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7229512] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229512/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229512/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52a7bd5cc42d44b7a2ec048619ff9f75 Command failed on smithi139 with status 5: 'rm -f /tmp/linux-image.deb && echo linux- image-6.3.0-rc3-g6c26fb6d76aa_6.3.0-rc3-g6c26fb6d76aa-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testi ng/6c26fb6d76aa35f265ac42f88c0a40a9c564d650/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-rc3-g6c26fb6d76aa/ --input-file=-' [7229516] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:08:09 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229516/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229516/ sentry: https://sentry.ceph.com/organizations/ceph/?query=275b2cd111be45eb95009167e0372efb HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229517] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229517/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229517/ sentry: https://sentry.ceph.com/organizations/ceph/?query=264880f132924dedb814da8a18b2c8a8 Command failed on smithi182 with status 5: 'rm -f /tmp/linux-image.deb && echo linux- image-6.3.0-rc3-g6c26fb6d76aa_6.3.0-rc3-g6c26fb6d76aa-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testi ng/6c26fb6d76aa35f265ac42f88c0a40a9c564d650/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-rc3-g6c26fb6d76aa/ --input-file=-' [7229518] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:07:57 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229518/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229518/ sentry: https://sentry.ceph.com/organizations/ceph/?query=048e8a80c5f6434d9ac9c3f64f3d0a10 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229524] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:34:25 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229524/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229524/ sentry: https://sentry.ceph.com/organizations/ceph/?query=614582ef0ca84da28b77a2e090d991ff Command failed (workunit test suites/pjd.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5a0c16ec27f924324bcd2d9df6a44607c0a3b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7229526] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} ----------------------------------------------------------------- time: 00:08:11 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229526/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229526/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3a751ece8b14f6cb94764b2b20b2742 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229539] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:07:44 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229539/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229539/ sentry: https://sentry.ceph.com/organizations/ceph/?query=029ea48436474f5496c9d1d4be4442ac HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229547] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:07:55 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229547/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229547/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df0a33d4c4df45dabbca24fa6d8f6861 HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/reef/c5a0c16ec27f924324bcd2d9df6a44607c0a3b4 d/ubuntu/jammy/flavors/default/repo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1108)'))) [7229559] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229559/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229559/ sentry: https://sentry.ceph.com/organizations/ceph/?query=97785c368c384cc49503f87c14173b55 Command failed on smithi007 with status 5: 'rm -f /tmp/linux-image.deb && echo linux- image-6.3.0-rc3-g6c26fb6d76aa_6.3.0-rc3-g6c26fb6d76aa-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testi ng/6c26fb6d76aa35f265ac42f88c0a40a9c564d650/ubuntu/focal/flavors/default/po ol/main/l/linux-6.3.0-rc3-g6c26fb6d76aa/ --input-file=-' Dead ================================================================= [7229425] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229425/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229425/ hit max job timeout [7229501] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229501/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229501/ hit max job timeout Pass ================================================================= [7229358] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:30:28 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229358/ [7229359] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:21:57 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229359/ [7229360] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:16 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229360/ [7229361] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:19:10 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229361/ [7229363] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:42 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229363/ [7229364] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:13:11 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229364/ [7229365] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:13 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229365/ [7229367] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:18:07 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229367/ [7229369] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:42 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229369/ [7229370] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:43 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229370/ [7229371] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:14:59 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229371/ [7229372] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:33:58 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229372/ [7229373] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:24:59 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229373/ [7229374] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:14:06 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229374/ [7229375] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:50:02 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229375/ [7229376] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:22 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229376/ [7229377] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:49 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229377/ [7229378] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:36:09 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229378/ [7229379] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:18:38 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229379/ [7229382] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:57:14 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229382/ [7229383] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:10:15 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229383/ [7229384] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:25:27 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229384/ [7229386] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:07:12 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229386/ [7229389] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:10 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229389/ [7229390] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:13:31 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229390/ [7229391] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:09 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229391/ [7229392] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:27:20 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229392/ [7229393] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:20:35 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229393/ [7229394] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} time: 02:25:57 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229394/ [7229395] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:46 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229395/ [7229396] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:14:24 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229396/ [7229397] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:14:06 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229397/ [7229398] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:12:49 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229398/ [7229399] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:53:04 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229399/ [7229400] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:27:34 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229400/ [7229403] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:17:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229403/ [7229405] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:19 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229405/ [7229406] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:14:37 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229406/ [7229407] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:48:48 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229407/ [7229409] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:20:36 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229409/ [7229413] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} time: 00:20:57 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229413/ [7229414] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:40:33 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229414/ [7229416] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:33:48 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229416/ [7229417] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:58 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229417/ [7229418] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:44:19 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229418/ [7229419] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:01:17 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229419/ [7229420] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/fstop} time: 00:14:42 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229420/ [7229421] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:17 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229421/ [7229422] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:31 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229422/ [7229423] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} time: 02:10:52 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229423/ [7229426] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:12:52 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229426/ [7229427] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:22 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229427/ [7229428] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:35 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229428/ [7229429] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:41:18 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229429/ [7229430] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:19:22 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229430/ [7229432] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:12:51 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229432/ [7229433] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:07:27 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229433/ [7229434] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:49 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229434/ [7229435] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:28:42 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229435/ [7229436] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:13:48 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229436/ [7229437] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:53:11 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229437/ [7229438] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:48:23 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229438/ [7229439] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:15 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229439/ [7229440] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:18 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229440/ [7229441] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 00:57:58 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229441/ [7229443] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:29 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229443/ [7229444] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:41 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229444/ [7229446] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:20:58 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229446/ [7229448] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:51:55 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229448/ [7229449] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:54:34 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229449/ [7229450] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:34:01 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229450/ [7229452] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:41:48 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229452/ [7229453] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:47:55 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229453/ [7229454] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:22:58 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229454/ [7229456] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} time: 01:15:04 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229456/ [7229458] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:12:12 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229458/ [7229459] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:27:56 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229459/ [7229460] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:23:48 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229460/ [7229461] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:19:33 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229461/ [7229463] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:13:24 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229463/ [7229466] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:16:46 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229466/ [7229467] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:07 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229467/ [7229468] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:16:47 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229468/ [7229469] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:29:43 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229469/ [7229470] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:35 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229470/ [7229471] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:13:45 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229471/ [7229472] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:36:40 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229472/ [7229473] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:19:31 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229473/ [7229478] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:59:28 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229478/ [7229481] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:13:56 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229481/ [7229482] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:36:23 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229482/ [7229483] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:20:16 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229483/ [7229484] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:24 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229484/ [7229485] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:44:01 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229485/ [7229488] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:41 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229488/ [7229489] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:13:36 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229489/ [7229490] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:21 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229490/ [7229492] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:16:54 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229492/ [7229493] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:35 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229493/ [7229494] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:37:48 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229494/ [7229495] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:23:00 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229495/ [7229496] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:18:29 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229496/ [7229497] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:49 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229497/ [7229498] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:16:22 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229498/ [7229499] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:17 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229499/ [7229502] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:53:29 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229502/ [7229503] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:09 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229503/ [7229504] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:20:40 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229504/ [7229508] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:52:40 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229508/ [7229509] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:41 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229509/ [7229511] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:58:41 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229511/ [7229513] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:25:34 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229513/ [7229514] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:54 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229514/ [7229515] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:30:47 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229515/ [7229519] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:40:38 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229519/ [7229520] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:52 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229520/ [7229521] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:12:57 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229521/ [7229522] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:19:29 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229522/ [7229523] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:38:12 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229523/ [7229525] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:13:39 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229525/ [7229527] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:13:41 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229527/ [7229528] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:19:25 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229528/ [7229531] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:17:39 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229531/ [7229533] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:43 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229533/ [7229535] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:55 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229535/ [7229537] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:33 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229537/ [7229541] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:16:43 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229541/ [7229543] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:28:58 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229543/ [7229545] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:13:38 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229545/ [7229549] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:25:12 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229549/ [7229551] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:36:30 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229551/ [7229553] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:20:15 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229553/ [7229555] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:24 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229555/ [7229557] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:13:20 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229557/ [7229561] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:27:53 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229561/ [7229562] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:34:41 info: http://pulpito.ceph.com/yuriw-2023-03-31_20:23:39-fs-reef-distro-default-smithi/7229562/