2023-05-09T05:51:16.985 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-09T05:51:17.002 INFO:teuthology.results:subset: '143/1024' 2023-05-09T05:51:17.002 INFO:teuthology.results:seed: '277' 2023-05-09T06:38:20.451 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-09T06:38:20.494 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-09T18:38:19.139 INFO:teuthology.results:Tests finished! gathering results... 2023-05-09T18:38:23.888 INFO:scrape:Found 54 jobs 2023-05-09T18:38:43.292 INFO:scrape:Found 13 distinct failure reasons 2023-05-09T18:38:43.294 INFO:scrape:Dead: b'2023-05-09T18:01:10.426 DEBUG:teuthology.exit:Finished running handlers' 2023-05-09T18:38:43.294 INFO:scrape:4 jobs: ['7267807', '7267828', '7267814', '7267844'] 2023-05-09T18:38:43.295 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-05-09T18:38:43.295 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/postgres}}', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/fscrypt-common}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-05-09T18:38:43.295 INFO:scrape: 2023-05-09T18:38:43.296 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi184 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench' 2023-05-09T18:38:43.296 INFO:scrape:4 jobs: ['7267818', '7267829', '7267854', '7267838'] 2023-05-09T18:38:43.297 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}'] 2023-05-09T18:38:43.297 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-dbench}', 'tasks/fscrypt-ffsb}', 'tasks/fscrypt-iozone}', 'tasks/fscrypt-pjd}'] 2023-05-09T18:38:43.297 INFO:scrape: 2023-05-09T18:38:43.298 INFO:scrape:Failure: Command failed on smithi123 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-05-09T18:38:43.298 INFO:scrape:4 jobs: ['7267815', '7267825', '7267847', '7267803'] 2023-05-09T18:38:43.298 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-05-09T18:38:43.299 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-05-09T18:38:43.299 INFO:scrape: 2023-05-09T18:38:43.299 INFO:scrape:Failure: "1683614793.8539882 mon.a (mon.0) 1046 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log 2023-05-09T18:38:43.300 INFO:scrape:2 jobs: ['7267830', '7267855'] 2023-05-09T18:38:43.300 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health'] 2023-05-09T18:38:43.300 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/client-recovery}', 'tasks/snap-schedule}'] 2023-05-09T18:38:43.301 INFO:scrape: 2023-05-09T18:38:43.301 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-05-09T18:38:43.302 INFO:scrape:1 jobs: ['7267848'] 2023-05-09T18:38:43.302 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}'] 2023-05-09T18:38:43.302 INFO:scrape: 2023-05-09T18:38:43.303 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-05-09T18:38:43.303 INFO:scrape:2 jobs: ['7267833', '7267850'] 2023-05-09T18:38:43.303 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}'] 2023-05-09T18:38:43.304 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-09T18:38:43.304 INFO:scrape: 2023-05-09T18:38:43.304 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' 2023-05-09T18:38:43.304 INFO:scrape:1 jobs: ['7267817'] 2023-05-09T18:38:43.305 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-05-09T18:38:43.305 INFO:scrape: 2023-05-09T18:38:43.306 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-05-09T18:38:43.306 INFO:scrape:2 jobs: ['7267849', '7267804'] 2023-05-09T18:38:43.306 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-05-09T18:38:43.307 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/quota}'] 2023-05-09T18:38:43.307 INFO:scrape: 2023-05-09T18:38:43.307 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-05-09T18:38:43.308 INFO:scrape:2 jobs: ['7267819', '7267841'] 2023-05-09T18:38:43.308 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-05-09T18:38:43.308 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-05-09T18:38:43.309 INFO:scrape: 2023-05-09T18:38:43.309 INFO:scrape:Failure: Command failed on smithi159 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2355a8b6-ee31-11ed-9b02-001a4aab830c -- ceph mon dump -f json' 2023-05-09T18:38:43.309 INFO:scrape:1 jobs: ['7267811'] 2023-05-09T18:38:43.309 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-05-09T18:38:43.310 INFO:scrape: 2023-05-09T18:38:43.310 INFO:scrape:Failure: Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) 2023-05-09T18:38:43.310 INFO:scrape:1 jobs: ['7267809'] 2023-05-09T18:38:43.311 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-05-09T18:38:43.311 INFO:scrape: 2023-05-09T18:38:43.311 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-05-09T18:38:43.311 INFO:scrape:1 jobs: ['7267851'] 2023-05-09T18:38:43.312 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-05-09T18:38:43.312 INFO:scrape: 2023-05-09T18:38:43.312 INFO:scrape:Failure: Command failed on smithi141 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 4b086ba51b906a75bac7252b2239e8f120a38c8c' 2023-05-09T18:38:43.313 INFO:scrape:1 jobs: ['7267806'] 2023-05-09T18:38:43.313 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2023-05-09T18:38:43.314 INFO:scrape: 2023-05-09T18:38:43.318 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/ failed: 22 dead: 4 running: 0 waiting: 0 queued: 0 passed: 28 Fail ================================================================= [7267803] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:15:33 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267803/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267803/ sentry: https://sentry.ceph.com/organizations/ceph/?query=26ebcd9b64f34f5097e930fe7ed6c0f2 Command failed on smithi082 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7267804] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:21:11 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267804/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267804/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19f87e956e0a4d14883c662440aa0f7f Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7267806] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:16:23 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267806/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01c4ad9b2cb74544bf475414096147dd Command failed on smithi141 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 4b086ba51b906a75bac7252b2239e8f120a38c8c' [7267809] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:17:50 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267809/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267809/ sentry: https://sentry.ceph.com/organizations/ceph/?query=14180b80947a480ea0409fe3c678edb1 Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) [7267811] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:12:59 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267811/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267811/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c7d9d42cf6f413ba40fd3c10108d70d Command failed on smithi159 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2355a8b6-ee31-11ed-9b02-001a4aab830c -- ceph mon dump -f json' [7267815] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:54 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267815/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267815/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9fa987066a1a4463b0d0f6579559d7ff Command failed on smithi123 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7267817] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:18:08 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267817/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267817/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0cc9b2e992743bfb6eb21fc730c2f3a Command failed (workunit test fs/quota/quota.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh' [7267818] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 00:09:36 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267818/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267818/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5aee9dc980e54c53abb07e57f2f0ae37 Command failed (workunit test fs/fscrypt.sh) on smithi184 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench' [7267819] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 06:32:16 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267819/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64ca63006571486292a71558f78c0c21 Command failed (workunit test suites/fsstress.sh) on smithi070 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7267825] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:15:23 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267825/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267825/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d9b860270ef4c2d89dbb5c69fea3dbf Command failed on smithi181 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7267829] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} ----------------------------------------------------------------- time: 00:09:44 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267829/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267829/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b4a9e9d10fa4301a67651490bbae922 Command failed (workunit test fs/fscrypt.sh) on smithi067 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none ffsb' [7267830] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:37:28 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267830/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267830/ "1683614793.8539882 mon.a (mon.0) 1046 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log [7267833] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:50:11 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267833/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267833/ sentry: https://sentry.ceph.com/organizations/ceph/?query=942d726716934d9baeefd5e6b00ebf54 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7267838] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} ----------------------------------------------------------------- time: 00:16:18 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267838/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267838/ sentry: https://sentry.ceph.com/organizations/ceph/?query=122c0f64d0c4470d93d0390d23ecaa6d Command failed (workunit test fs/fscrypt.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none iozone' [7267841] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 06:43:59 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267841/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff4a65665b0d4840a613ac37ac5e5935 Command failed (workunit test suites/fsstress.sh) on smithi038 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7267847] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:27 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267847/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267847/ sentry: https://sentry.ceph.com/organizations/ceph/?query=115d9acc2e5946eca54954323ff0e92d Command failed on smithi176 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7267848] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:16:08 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267848/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267848/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d874d9e43a9447d9b60539f10cfb2a7 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7267849] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:15:12 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267849/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51cedb2bcb1247238c61c44146dc79ba Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7267850] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:55:25 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267850/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267850/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2684c2c8292846b998f8859b52bc9c03 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7267851] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:21:28 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267851/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267851/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4f2a68535534041bd3668deffbd564c Command failed (workunit test fs/test_python.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7267854] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} ----------------------------------------------------------------- time: 00:10:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267854/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab40930e54cd4163bf712dba293bcdb8 Command failed (workunit test fs/fscrypt.sh) on smithi191 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b086ba51b906a75bac7252b2239e8f120a38c8c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none pjd' [7267855] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:51:49 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267855/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267855/ "1683615688.0045855 mon.a (mon.0) 461 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log Dead ================================================================= [7267807] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267807/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267807/ hit max job timeout [7267814] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267814/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267814/ hit max job timeout [7267828] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267828/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267828/ hit max job timeout [7267844] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267844/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267844/ hit max job timeout Pass ================================================================= [7267802] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 02:23:44 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267802/ [7267805] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:29:48 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267805/ [7267808] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit} time: 00:28:19 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267808/ [7267810] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:35:56 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267810/ [7267812] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:54 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267812/ [7267813] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:16 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267813/ [7267816] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:54:40 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267816/ [7267820] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:15:51 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267820/ [7267821] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:11:17 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267821/ [7267822] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:30:49 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267822/ [7267823] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:35:31 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267823/ [7267824] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:13 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267824/ [7267826] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:40:05 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267826/ [7267827] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:44:21 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267827/ [7267831] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:27:11 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267831/ [7267832] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 01:00:07 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267832/ [7267834] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:23:43 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267834/ [7267835] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:51:31 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267835/ [7267836] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}} time: 00:07:49 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267836/ [7267837] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:10:37 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267837/ [7267839] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:17:01 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267839/ [7267840] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:29:30 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267840/ [7267842] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:08 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267842/ [7267843] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:37 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267843/ [7267845] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:17:23 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267845/ [7267846] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:35 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267846/ [7267852] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:56 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267852/ [7267853] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:37:02 info: http://pulpito.ceph.com/vshankar-2023-05-09_05:47:12-fs-wip-vshankar-testing-20230506.143554-testing-default-smithi/7267853/