2023-03-28T09:52:44.026 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-28T09:52:44.027 INFO:teuthology.results:subset: '111/120000' 2023-03-28T09:52:44.027 INFO:teuthology.results:seed: '4984' 2023-03-28T14:32:13.695 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-28T14:32:13.720 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-28T23:04:01.160 INFO:teuthology.results:Tests finished! gathering results... 2023-03-28T23:04:18.927 INFO:scrape:Found 211 jobs 2023-03-28T23:04:35.777 WARNING:scrape:Misunderstood line: 2023-03-28T23:04:39.552 INFO:scrape:Found 13 distinct failure reasons 2023-03-28T23:04:39.553 INFO:scrape:Failure: "1680007445.3916788 mon.a (mon.0) 1091 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log 2023-03-28T23:04:39.553 INFO:scrape:2 jobs: ['7222330', '7222331'] 2023-03-28T23:04:39.554 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health'] 2023-03-28T23:04:39.554 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}', 'tasks/failover}'] 2023-03-28T23:04:39.554 INFO:scrape: 2023-03-28T23:04:39.555 INFO:scrape:Failure: local variable 'expected_val' referenced before assignment 2023-03-28T23:04:39.555 INFO:scrape:13 jobs: ['7222336', '7222310', '7222258', '7222369', '7222294', '7222391', '7222240', '7222379', '7222205', '7222195', '7222215', '7222346', '7222357'] 2023-03-28T23:04:39.556 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2023-03-28T23:04:39.556 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iozone}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2023-03-28T23:04:39.556 INFO:scrape: 2023-03-28T23:04:39.557 INFO:scrape:Failure: Command failed on smithi045 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-03-28T23:04:39.557 INFO:scrape:5 jobs: ['7222295', '7222301', '7222262', '7222189', '7222366'] 2023-03-28T23:04:39.557 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-03-28T23:04:39.558 INFO:scrape:suites union: ['1-ceph', '1-client', '2-logrotate}', '2-upgrade', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client', 'clusters/1-mds-2-client-micro', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'multimds/yes', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/{0-octopus'] 2023-03-28T23:04:39.558 INFO:scrape: 2023-03-28T23:04:39.558 INFO:scrape:Dead: b'2023-03-28T22:29:12.625 DEBUG:teuthology.exit:Finished running handlers' 2023-03-28T23:04:39.558 INFO:scrape:3 jobs: ['7222220', '7222235', '7222250'] 2023-03-28T23:04:39.559 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}'] 2023-03-28T23:04:39.559 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '2-workunit/cfuse_workunit_snaptests}}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/iogen}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'msgr-failures/osd-mds-delay', 'multifs', 'n/5', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/fscrypt-common}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'wsync/no}'] 2023-03-28T23:04:39.559 INFO:scrape: 2023-03-28T23:04:39.560 INFO:scrape:Failure: No module named 'tasks' 2023-03-28T23:04:39.560 INFO:scrape:5 jobs: ['7222312', '7222299', '7222287', '7222302', '7222313'] 2023-03-28T23:04:39.560 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-03-28T23:04:39.561 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-workunit/suites/fsstress}}', '3-snaps/no', '4-flush/yes', '5-workunit/direct_io}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'msgr-failures/none', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'rhel_8}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/fscrypt-ffsb}', 'tasks/pool-perm}', 'tasks/quota}', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'wsync/yes}'] 2023-03-28T23:04:39.561 INFO:scrape: 2023-03-28T23:04:39.561 INFO:scrape:Failure: Command failed on smithi141 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 9d12d2cb27d4a45c58218390bfbb56820b010b77' 2023-03-28T23:04:39.562 INFO:scrape:1 jobs: ['7222285'] 2023-03-28T23:04:39.562 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/no}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2023-03-28T23:04:39.562 INFO:scrape: 2023-03-28T23:04:39.562 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi129 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d12d2cb27d4a45c58218390bfbb56820b010b77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-03-28T23:04:39.563 INFO:scrape:1 jobs: ['7222385'] 2023-03-28T23:04:39.563 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-03-28T23:04:39.563 INFO:scrape: 2023-03-28T23:04:39.564 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2023-03-28T23:04:39.564 INFO:scrape:1 jobs: ['7222320'] 2023-03-28T23:04:39.564 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}'] 2023-03-28T23:04:39.564 INFO:scrape: 2023-03-28T23:04:39.565 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d12d2cb27d4a45c58218390bfbb56820b010b77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-03-28T23:04:39.565 INFO:scrape:1 jobs: ['7222274'] 2023-03-28T23:04:39.565 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}'] 2023-03-28T23:04:39.566 INFO:scrape: 2023-03-28T23:04:39.566 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-03-28T23:04:39.566 INFO:scrape:1 jobs: ['7222316'] 2023-03-28T23:04:39.567 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-03-28T23:04:39.567 INFO:scrape: 2023-03-28T23:04:39.567 INFO:scrape:Failure: Config file not found: "/home/teuthworker/src/github.com_ceph_ceph-c_9d12d2cb27d4a45c58218390bfbb56820b010b77/qa/tasks/ceph.conf.template". 2023-03-28T23:04:39.567 INFO:scrape:1 jobs: ['7222297'] 2023-03-28T23:04:39.568 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-03-28T23:04:39.568 INFO:scrape: 2023-03-28T23:04:39.568 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi170 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d12d2cb27d4a45c58218390bfbb56820b010b77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-03-28T23:04:39.569 INFO:scrape:1 jobs: ['7222343'] 2023-03-28T23:04:39.569 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-03-28T23:04:39.569 INFO:scrape: 2023-03-28T23:04:39.569 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-03-28T23:04:39.570 INFO:scrape:1 jobs: ['7222355'] 2023-03-28T23:04:39.570 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-03-28T23:04:39.570 INFO:scrape: 2023-03-28T23:04:39.583 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/ failed: 33 dead: 3 running: 0 waiting: 0 queued: 0 passed: 175 Fail ================================================================= [7222189] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:28 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222189/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3452637905e94de49e2e93df2e321c0b Command failed on smithi078 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7222195] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:35:38 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222195/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222195/ local variable 'expected_val' referenced before assignment [7222205] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:33:06 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222205/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222205/ local variable 'expected_val' referenced before assignment [7222215] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:36:27 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222215/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222215/ local variable 'expected_val' referenced before assignment [7222240] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:36:25 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222240/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222240/ local variable 'expected_val' referenced before assignment [7222258] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:28:12 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222258/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222258/ local variable 'expected_val' referenced before assignment [7222262] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:41 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222262/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222262/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e4c222a96294beb85939245fc4284d0 Command failed on smithi122 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7222274] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:40 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222274/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222274/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5530ade10e324435991d90fd4e5b807d Command failed (workunit test suites/pjd.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d12d2cb27d4a45c58218390bfbb56820b010b77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7222285] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:25:41 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222285/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222285/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67955479575343b899139ac76a304978 Command failed on smithi141 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 9d12d2cb27d4a45c58218390bfbb56820b010b77' [7222287] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:25:06 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222287/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222287/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b13425cf772444db72b1a222a572e71 No module named 'tasks.fs' [7222294] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:11:09 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222294/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222294/ local variable 'expected_val' referenced before assignment [7222295] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:17:15 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222295/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222295/ sentry: https://sentry.ceph.com/organizations/ceph/?query=36b4dfde83674bf4b28a6aca3cd402e7 Command failed on smithi045 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7222297] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:11:06 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222297/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222297/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca8c894b364641d2bc10e8a1a1648fca Config file not found: "/home/teuthworker/src/github.com_ceph_ceph- c_9d12d2cb27d4a45c58218390bfbb56820b010b77/qa/tasks/ceph.conf.template". [7222299] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:13:36 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222299/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222299/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9df103da86f7472da2efd14a27636b9a No module named 'tasks.ceph_fuse' [7222301] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:09:03 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222301/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222301/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dfb32f7cae784cb99859f30661459d57 Command failed on smithi188 with status 1: 'sudo yum -y install cephfs- java' [7222302] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:09:34 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222302/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222302/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fed6d3e9bcff46cb8741c5cecabff01d No module named 'tasks.ceph' [7222310] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:42:44 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222310/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222310/ local variable 'expected_val' referenced before assignment [7222312] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} ----------------------------------------------------------------- time: 00:13:51 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222312/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222312/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4702d336b907482da92b3e59040ce8f8 No module named 'tasks' [7222313] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:09:46 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222313/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222313/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e4e06cf29ac4cc8b163ea82bdd4e8b4 No module named 'tasks' [7222316] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 01:57:54 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222316/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222316/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd886d06d6514bb3942698091ee6fb65 saw valgrind issues [7222320] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:30:23 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222320/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222320/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7222330] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 01:03:14 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222330/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222330/ "1680007445.3916788 mon.a (mon.0) 1091 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7222331] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:54:54 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222331/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222331/ "1680008144.9130812 mon.a (mon.0) 3541 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7222336] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:42:20 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222336/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222336/ local variable 'expected_val' referenced before assignment [7222343] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:36:30 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222343/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=05ed2a9235a44779bd8489b517d09ce2 Command failed (workunit test kernel_untar_build.sh) on smithi170 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d12d2cb27d4a45c58218390bfbb56820b010b77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7222346] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:12:29 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222346/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222346/ local variable 'expected_val' referenced before assignment [7222355] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 07:42:55 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222355/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222355/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f429d0eba054d78ae47a8c1dd92c2ed Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7222357] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:42:39 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222357/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222357/ local variable 'expected_val' referenced before assignment [7222366] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:20:22 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222366/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222366/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1289445706b74273bc38c0a552d304d0 Command failed on smithi141 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7222369] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:35:22 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222369/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222369/ local variable 'expected_val' referenced before assignment [7222379] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:29:30 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222379/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222379/ local variable 'expected_val' referenced before assignment [7222385] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:43:55 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222385/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222385/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cae5d49a3bd6416692c063a2638157a9 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi129 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d12d2cb27d4a45c58218390bfbb56820b010b77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7222391] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:38:34 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222391/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222391/ local variable 'expected_val' referenced before assignment Dead ================================================================= [7222220] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222220/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222220/ hit max job timeout [7222235] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222235/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222235/ hit max job timeout [7222250] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222250/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222250/ hit max job timeout Pass ================================================================= [7222185] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/blogbench}} time: 00:23:55 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222185/ [7222186] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:02 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222186/ [7222187] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:19:13 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222187/ [7222188] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:18:12 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222188/ [7222190] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:50:33 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222190/ [7222191] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:59:03 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222191/ [7222192] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:22:45 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222192/ [7222193] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:38 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222193/ [7222194] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:13:54 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222194/ [7222196] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:46 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222196/ [7222197] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:51 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222197/ [7222198] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/dbench}} time: 00:53:12 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222198/ [7222199] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:25:10 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222199/ [7222200] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:13:15 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222200/ [7222201] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/fsstress}} time: 00:18:51 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222201/ [7222202] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:28:07 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222202/ [7222203] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:55:52 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222203/ [7222204] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:41 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222204/ [7222206] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:14:10 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222206/ [7222207] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:12:45 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222207/ [7222208] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:15 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222208/ [7222209] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:20:00 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222209/ [7222210] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:13:08 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222210/ [7222211] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/iozone}} time: 00:16:40 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222211/ [7222212] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:49:54 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222212/ [7222213] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:16:11 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222213/ [7222214] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}} time: 01:19:21 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222214/ [7222216] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:22:24 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222216/ [7222217] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:46 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222217/ [7222218] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:14:06 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222218/ [7222219] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:48:17 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222219/ [7222221] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:20:51 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222221/ [7222222] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:14:49 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222222/ [7222223] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 02:04:23 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222223/ [7222224] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit} time: 00:21:13 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222224/ [7222225] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:42:40 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222225/ [7222226] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:04:56 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222226/ [7222227] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:35:11 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222227/ [7222228] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:08 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222228/ [7222229] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:57:02 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222229/ [7222230] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:09:18 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222230/ [7222231] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/fstop} time: 00:14:38 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222231/ [7222232] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:11 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222232/ [7222233] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:28 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222233/ [7222234] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} time: 02:13:45 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222234/ [7222236] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:12:42 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222236/ [7222237] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:38 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222237/ [7222238] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:54:22 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222238/ [7222239] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:18:45 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222239/ [7222241] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:12:48 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222241/ [7222242] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:40:48 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222242/ [7222243] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/new_ops/client-sanity}} time: 00:12:43 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222243/ [7222244] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:22:44 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222244/ [7222245] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:52 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222245/ [7222246] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:24:29 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222246/ [7222247] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/blogbench}} time: 00:24:02 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222247/ [7222248] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:36:42 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222248/ [7222249] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:13:42 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222249/ [7222251] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:34 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222251/ [7222252] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:50:51 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222252/ [7222253] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/dbench}} time: 00:55:38 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222253/ [7222254] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:32 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222254/ [7222255] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:52 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222255/ [7222256] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:53 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222256/ [7222257] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:28:02 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222257/ [7222259] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:19:58 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222259/ [7222260] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:18:43 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222260/ [7222261] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/fsstress}} time: 00:17:59 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222261/ [7222263] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:53:34 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222263/ [7222264] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:56:14 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222264/ [7222265] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:23:34 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222265/ [7222266] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:54 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222266/ [7222267] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:42:04 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222267/ [7222268] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:52:47 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222268/ [7222269] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:19:47 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222269/ [7222270] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} time: 01:15:59 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222270/ [7222271] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:36:09 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222271/ [7222272] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:11:43 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222272/ [7222273] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/iozone}} time: 00:16:48 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222273/ [7222275] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:03 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222275/ [7222276] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:18:24 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222276/ [7222277] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:19 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222277/ [7222278] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:23:52 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222278/ [7222279] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}} time: 01:16:10 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222279/ [7222280] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:47 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222280/ [7222281] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:19:29 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222281/ [7222282] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:13:52 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222282/ [7222283] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:43 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222283/ [7222284] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:51:12 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222284/ [7222286] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:37:33 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222286/ [7222288] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 00:50:56 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222288/ [7222289] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/new_ops/client-sanity}} time: 00:13:21 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222289/ [7222290] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:18:48 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222290/ [7222291] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:16:43 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222291/ [7222292] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:16:16 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222292/ [7222293] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/blogbench}} time: 00:24:27 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222293/ [7222296] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:28:59 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222296/ [7222298] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:42:34 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222298/ [7222300] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:45:41 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222300/ [7222303] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/dbench}} time: 00:54:52 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222303/ [7222304] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:59 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222304/ [7222305] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:46:47 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222305/ [7222306] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:56 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222306/ [7222307] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:37:14 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222307/ [7222308] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:29:51 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222308/ [7222309] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:14:20 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222309/ [7222311] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:53 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222311/ [7222314] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/fsstress}} time: 00:19:06 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222314/ [7222315] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:24:36 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222315/ [7222317] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:27:17 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222317/ [7222318] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:16:15 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222318/ [7222319] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:27 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222319/ [7222321] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/iozone}} time: 00:16:06 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222321/ [7222322] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 00:54:56 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222322/ [7222323] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:09:39 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222323/ [7222324] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:14:10 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222324/ [7222325] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}} time: 01:18:22 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222325/ [7222326] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:19:24 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222326/ [7222327] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:25 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222327/ [7222328] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:19:45 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222328/ [7222329] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:15:32 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222329/ [7222332] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:40 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222332/ [7222333] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:14:49 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222333/ [7222334] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:25 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222334/ [7222335] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:26:11 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222335/ [7222337] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:14 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222337/ [7222338] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:35:38 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222338/ [7222339] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:18:23 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222339/ [7222340] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:13:13 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222340/ [7222341] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:20:41 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222341/ [7222342] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/new_ops/client-sanity}} time: 00:13:05 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222342/ [7222344] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:18:37 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222344/ [7222345] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:18:09 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222345/ [7222347] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:12:51 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222347/ [7222348] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/blogbench}} time: 00:23:54 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222348/ [7222349] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:18:06 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222349/ [7222350] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:27 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222350/ [7222351] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:18:29 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222351/ [7222352] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} time: 02:36:57 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222352/ [7222353] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:12:59 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222353/ [7222354] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:12 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222354/ [7222356] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} time: 01:24:53 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222356/ [7222358] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:23:59 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222358/ [7222359] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:19:38 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222359/ [7222360] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:47:49 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222360/ [7222361] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/dbench}} time: 00:55:13 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222361/ [7222362] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:14:54 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222362/ [7222363] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/fsstress}} time: 00:17:43 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222363/ [7222364] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:43:38 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222364/ [7222365] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:13:34 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222365/ [7222367] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:26:25 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222367/ [7222368] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:39:38 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222368/ [7222370] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:16:37 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222370/ [7222371] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:17:59 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222371/ [7222372] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:28:16 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222372/ [7222373] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:13:17 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222373/ [7222374] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/iozone}} time: 00:16:32 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222374/ [7222375] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:40:50 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222375/ [7222376] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:21:07 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222376/ [7222377] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:23:07 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222377/ [7222378] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}} time: 01:16:53 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222378/ [7222380] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:32 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222380/ [7222381] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:13:16 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222381/ [7222382] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:30:52 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222382/ [7222383] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:42 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222383/ [7222384] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:25:45 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222384/ [7222386] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:16:49 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222386/ [7222387] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:47 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222387/ [7222388] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:49 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222388/ [7222389] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:17:24 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222389/ [7222390] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:23:03 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222390/ [7222392] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:22:23 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222392/ [7222393] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/new_ops/client-sanity}} time: 00:12:50 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222393/ [7222394] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:12:44 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222394/ [7222395] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:54 info: http://pulpito.ceph.com/yuriw-2023-03-28_00:15:34-fs-wip-yuri5-testing-2023-03-27-1315-distro-default-smithi/7222395/