2024-04-13T02:45:56.585 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-13T02:45:56.586 INFO:teuthology.results:subset: '55/1024' 2024-04-13T02:45:56.586 INFO:teuthology.results:seed: '7517' 2024-04-13T06:13:31.123 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-13T06:13:31.159 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-13T17:17:31.413 INFO:teuthology.results:Tests finished! gathering results... 2024-04-13T17:17:52.068 INFO:scrape:Found 204 jobs 2024-04-13T17:18:39.438 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.438 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.439 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.439 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.566 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.566 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.566 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.566 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.640 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.640 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.641 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.641 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.718 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.719 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.719 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.719 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.793 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.794 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.794 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:39.794 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.267 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.268 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.268 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.268 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.350 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.350 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.350 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.350 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.436 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.436 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.436 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.436 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.513 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.514 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.514 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.514 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.595 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.595 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.595 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.595 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.671 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.671 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.671 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.671 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.745 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.745 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.745 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.746 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.824 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.824 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.824 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.824 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.906 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.906 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.906 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.906 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.981 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.981 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.981 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:42.981 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.056 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.056 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.057 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.057 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.134 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.134 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.134 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.134 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.208 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.209 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.209 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.209 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.287 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.287 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.287 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.288 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.371 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.371 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.371 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.371 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.447 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.447 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.447 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.448 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.521 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.521 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.521 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:43.521 WARNING:scrape:Saw end of BT but not start 2024-04-13T17:18:59.360 INFO:scrape:Found 20 distinct failure reasons 2024-04-13T17:18:59.360 INFO:scrape:Failure: Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2024-04-13T17:18:59.361 INFO:scrape:2 jobs: ['7654288', '7654279'] 2024-04-13T17:18:59.361 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'mount/fuse', 'overrides/{ignorelist_health', 'pg_health}'] 2024-04-13T17:18:59.361 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/{ignorelist_health', 'pg_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/mirror}', 'tasks/mirror}}'] 2024-04-13T17:18:59.361 INFO:scrape: 2024-04-13T17:18:59.361 INFO:scrape:Failure: "2024-04-13T05:24:46.284907+0000 mds.h (mds.1) 26 : cluster [WRN] Scrub error on inode 0x20000001a4d (/volumes/qa/sv_0/695f39a4-c9d5-4816-845e-3fe961904a07/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-44) see mds.h log and `damage ls` output for details" in cluster log 2024-04-13T17:18:59.361 INFO:scrape:1 jobs: ['7654333'] 2024-04-13T17:18:59.362 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/no', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}'] 2024-04-13T17:18:59.362 INFO:scrape: 2024-04-13T17:18:59.362 INFO:scrape:Failure: "2024-04-13T06:26:38.758122+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log 2024-04-13T17:18:59.362 INFO:scrape:3 jobs: ['7654424', '7654237', '7654283'] 2024-04-13T17:18:59.362 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg_health}'] 2024-04-13T17:18:59.362 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/nfs/{cluster/{1-node}', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'supported-random-distros$/{centos_latest}', 'tasks/forward-scrub}', 'tasks/nfs}'] 2024-04-13T17:18:59.363 INFO:scrape: 2024-04-13T17:18:59.363 INFO:scrape:Failure: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-04-13T17:18:59.363 INFO:scrape:2 jobs: ['7654262', '7654426'] 2024-04-13T17:18:59.363 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2024-04-13T17:18:59.363 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-04-13T17:18:59.363 INFO:scrape: 2024-04-13T17:18:59.363 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-13T17:18:59.363 INFO:scrape:3 jobs: ['7654242', '7654276', '7654340'] 2024-04-13T17:18:59.364 INFO:scrape:suites intersection: [] 2024-04-13T17:18:59.364 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-mount', '2-logrotate', '2-scrub/no', '2-workload/dbench', '3-modules}', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/single-container-host', 'export-check', 'frag', 'fs/cephadm/multivolume/{0-start', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}}', 'ranks/1', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}', 'wsync/yes}'] 2024-04-13T17:18:59.364 INFO:scrape: 2024-04-13T17:18:59.364 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2024-04-13T17:18:59.364 INFO:scrape:1 jobs: ['7654361'] 2024-04-13T17:18:59.364 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/acls}'] 2024-04-13T17:18:59.364 INFO:scrape: 2024-04-13T17:18:59.365 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2024-04-13T17:18:59.365 INFO:scrape:1 jobs: ['7654293'] 2024-04-13T17:18:59.365 INFO:scrape:suites: ['1-ceph', '2-logrotate', '2-workunit/cfuse_workunit_snaptests}}', '3-modules}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'pg_health', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2024-04-13T17:18:59.365 INFO:scrape: 2024-04-13T17:18:59.365 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-04-13T17:18:59.365 INFO:scrape:1 jobs: ['7654390'] 2024-04-13T17:18:59.365 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cephfs-shell}'] 2024-04-13T17:18:59.365 INFO:scrape: 2024-04-13T17:18:59.366 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-04-13T17:18:59.366 INFO:scrape:6 jobs: ['7654343', '7654289', '7654417', '7654243', '7654310', '7654374'] 2024-04-13T17:18:59.366 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}'] 2024-04-13T17:18:59.366 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-04-13T17:18:59.366 INFO:scrape: 2024-04-13T17:18:59.366 INFO:scrape:Failure: "2024-04-13T03:43:57.359428+0000 mds.i (mds.0) 64 : cluster [WRN] Scrub error on inode 0x100000004e4 (/volumes/qa/sv_1/2351a094-2ab6-4707-b8f6-e3b3a77041f3/client.0/tmp/tmp/data) see mds.i log and `damage ls` output for details" in cluster log 2024-04-13T17:18:59.366 INFO:scrape:1 jobs: ['7654251'] 2024-04-13T17:18:59.367 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-04-13T17:18:59.367 INFO:scrape: 2024-04-13T17:18:59.367 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-13T17:18:59.367 INFO:scrape:4 jobs: ['7654321', '7654305', '7654294', '7654260'] 2024-04-13T17:18:59.367 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'pg_health', 'ranks/multi/{balancer/automatic', 'session_timeout}', 'standby-replay'] 2024-04-13T17:18:59.367 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/iogen}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}', 'wsync/yes}'] 2024-04-13T17:18:59.367 INFO:scrape: 2024-04-13T17:18:59.368 INFO:scrape:Dead: b'2024-04-13T16:54:47.683 DEBUG:teuthology.exit:Finished running handlers' 2024-04-13T17:18:59.368 INFO:scrape:ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3302042520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x56255a9bb9f6] 6: (KernelDevice::_aio_thread()+0x9e2) [0x56255b371bd2] 7: ceph-osd(+0xd889b1) [0x56255b34d9b1] 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3302094b43] 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f3302126a00] 2024-04-13T17:18:59.368 INFO:scrape:1 jobs: ['7654350'] 2024-04-13T17:18:59.368 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-04-13T17:18:59.368 INFO:scrape: 2024-04-13T17:18:59.368 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-04-13T17:18:59.368 INFO:scrape:1 jobs: ['7654432'] 2024-04-13T17:18:59.369 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-04-13T17:18:59.369 INFO:scrape: 2024-04-13T17:18:59.369 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-13T17:18:59.369 INFO:scrape:1 jobs: ['7654408'] 2024-04-13T17:18:59.369 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-13T17:18:59.369 INFO:scrape: 2024-04-13T17:18:59.369 INFO:scrape:Failure: "2024-04-13T04:00:00.000209+0000 mon.smithi176 (mon.0) 351 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 46/375 objects degraded (12.267%), 13 pgs degraded" in cluster log 2024-04-13T17:18:59.369 INFO:scrape:2 jobs: ['7654263', '7654405'] 2024-04-13T17:18:59.370 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'pg_health'] 2024-04-13T17:18:59.370 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '2-logrotate', '2-scrub/yes', '3-inline/no', '3-modules}', '3-snaps/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-verify}', '5-upgrade-with-workload', '5-workunit/fs/misc}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'fail_fs/no', 'frag', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'ranks/multi/{balancer/random', 'replication/default}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-subvolume/{with-quota}'] 2024-04-13T17:18:59.370 INFO:scrape: 2024-04-13T17:18:59.370 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-04-13T17:18:59.370 INFO:scrape:1 jobs: ['7654415'] 2024-04-13T17:18:59.370 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2024-04-13T17:18:59.370 INFO:scrape: 2024-04-13T17:18:59.370 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi106 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-04-13T17:18:59.371 INFO:scrape:2 jobs: ['7654363', '7654266'] 2024-04-13T17:18:59.371 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'pg_health', 'ranks/multi/{balancer/random', 'session_timeout}', 'standby-replay'] 2024-04-13T17:18:59.371 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-modules}', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-04-13T17:18:59.371 INFO:scrape: 2024-04-13T17:18:59.371 INFO:scrape:Failure: "2024-04-13T05:49:52.911915+0000 mds.c (mds.0) 42 : cluster [WRN] client.4684 isn't responding to mclientcaps(revoke), ino 0x100000030b2 pending pAsxXsxFsxcrwb issued pAsxLsXsxFsxcrwb, sent 301.790873 seconds ago" in cluster log 2024-04-13T17:18:59.371 INFO:scrape:1 jobs: ['7654347'] 2024-04-13T17:18:59.371 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2024-04-13T17:18:59.372 INFO:scrape: 2024-04-13T17:18:59.372 INFO:scrape:Failure: Test failure: test_quiesce_authpin_wait (tasks.cephfs.test_quiesce.TestQuiesceMultiRank) 2024-04-13T17:18:59.372 INFO:scrape:1 jobs: ['7654296'] 2024-04-13T17:18:59.372 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/quiesce}'] 2024-04-13T17:18:59.372 INFO:scrape: 2024-04-13T17:18:59.372 INFO:scrape:Dead: b'2024-04-13T17:13:42.192 DEBUG:teuthology.exit:Finished running handlers' 2024-04-13T17:18:59.372 INFO:scrape:1 jobs: ['7654366'] 2024-04-13T17:18:59.372 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}'] 2024-04-13T17:18:59.372 INFO:scrape: 2024-04-13T17:18:59.387 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/ failed: 34 dead: 2 running: 0 waiting: 0 queued: 0 passed: 168 Fail ================================================================= [7654237] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} ----------------------------------------------------------------- time: 00:19:27 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654237/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654237/ "2024-04-13T03:15:35.247546+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log [7654242] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:47:39 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654242/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df04e36084ae4a3cb22dd023563280d6 Command failed (workunit test suites/dbench.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7654243] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:47:41 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654243/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654243/ reached maximum tries (51) after waiting for 300 seconds [7654251] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:49:57 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654251/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654251/ "2024-04-13T03:43:57.359428+0000 mds.i (mds.0) 64 : cluster [WRN] Scrub error on inode 0x100000004e4 (/volumes/qa/sv_1/2351a094-2ab6-4707-b8f6-e3b3 a77041f3/client.0/tmp/tmp/data) see mds.i log and `damage ls` output for details" in cluster log [7654260] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:26:11 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654260/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654260/ error during scrub thrashing: rank damage found: {'backtrace'} [7654262] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:17:11 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654262/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654262/ valgrind error: Leak_PossiblyLost calloc __trans_list_add [7654263] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:39:52 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654263/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654263/ "2024-04-13T04:00:00.000209+0000 mon.smithi176 (mon.0) 351 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 46/375 objects degraded (12.267%), 13 pgs degraded" in cluster log [7654266] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:36:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654266/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654266/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75ac8d10b3764146923020cf344f81c5 Command failed (workunit test suites/fsx.sh) on smithi086 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7654276] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host overrides/{ignorelist_health pg_health}} ----------------------------------------------------------------- time: 00:41:24 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654276/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654276/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc57165541504fb2b71a46d8040f0082 Command failed (workunit test suites/dbench.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7654279] fs/mirror/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} ----------------------------------------------------------------- time: 00:21:57 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654279/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654279/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0e7425ccba94fdf87d48979e58e8875 Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7654283] fs/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:45:14 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654283/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654283/ "2024-04-13T04:07:02.742020+0000 mon.a (mon.0) 330 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log [7654288] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/{ignorelist_health pg_health} tasks/mirror}} ----------------------------------------------------------------- time: 00:30:52 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654288/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654288/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27cae03480e748e79433c0baf9d89b27 Test failure: test_cephfs_mirror_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7654289] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:49:14 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654289/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654289/ reached maximum tries (51) after waiting for 300 seconds [7654293] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:36:57 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654293/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654293/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e6b2c29f817414691389b5c25c18905 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7654294] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:35:09 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654294/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654294/ error during scrub thrashing: rank damage found: {'backtrace'} [7654296] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quiesce} ----------------------------------------------------------------- time: 00:36:19 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654296/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654296/ sentry: https://sentry.ceph.com/organizations/ceph/?query=463dd8ce5d4645f18efcb27d2806331e Test failure: test_quiesce_authpin_wait (tasks.cephfs.test_quiesce.TestQuiesceMultiRank) [7654305] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:24:40 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654305/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654305/ error during scrub thrashing: rank damage found: {'backtrace'} [7654310] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654310/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654310/ reached maximum tries (51) after waiting for 300 seconds [7654321] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:10:12 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654321/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654321/ error during scrub thrashing: rank damage found: {'backtrace'} [7654333] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:42:15 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654333/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654333/ "2024-04-13T05:24:46.284907+0000 mds.h (mds.1) 26 : cluster [WRN] Scrub error on inode 0x20000001a4d (/volumes/qa/sv_0/695f39a4-c9d5-4816-845e-3fe9 61904a07/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-44) see mds.h log and `damage ls` output for details" in cluster log [7654340] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:34:34 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654340/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654340/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a221d9434f174f3aa32c0a9199ab2220 Command failed (workunit test suites/dbench.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7654343] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:36 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654343/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654343/ reached maximum tries (51) after waiting for 300 seconds [7654347] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:19:15 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654347/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654347/ "2024-04-13T05:49:52.911915+0000 mds.c (mds.0) 42 : cluster [WRN] client.4684 isn't responding to mclientcaps(revoke), ino 0x100000030b2 pending pAsxXsxFsxcrwb issued pAsxLsXsxFsxcrwb, sent 301.790873 seconds ago" in cluster log [7654361] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/acls} ----------------------------------------------------------------- time: 00:09:29 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654361/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654361/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ad5d457066b4edcb8151438cb169958 Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) [7654363] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:31:16 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654363/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654363/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5c39f7535964880ac1cbac66128699a Command failed (workunit test suites/fsx.sh) on smithi106 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7654374] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:48:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654374/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654374/ reached maximum tries (51) after waiting for 300 seconds [7654390] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:13:54 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654390/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2773bcc3ec5944a5944aa31e698ca506 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7654405] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:29:27 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654405/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654405/ "2024-04-13T06:10:00.000190+0000 mon.a (mon.0) 589 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded" in cluster log [7654408] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:55:26 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654408/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=350bb92d6c6744cb87e2305110ffdb80 Command failed (workunit test suites/fsstress.sh) on smithi181 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7654415] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:58:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654415/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654415/ The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} [7654417] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:47:53 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654417/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654417/ reached maximum tries (51) after waiting for 300 seconds [7654424] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} ----------------------------------------------------------------- time: 00:23:38 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654424/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654424/ "2024-04-13T06:26:38.758122+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log [7654426] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:36:24 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654426/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654426/ valgrind error: Leak_PossiblyLost calloc __trans_list_add [7654432] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:34:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654432/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654432/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db33f0be5a85464497037ff93ed2d483 Command failed (workunit test suites/ffsb.sh) on smithi016 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=218a691417ef57c5d22a8d5cf3c5f41f7b542838 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' Dead ================================================================= [7654350] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654350/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654350/ hit max job timeout [7654366] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654366/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654366/ hit max job timeout Pass ================================================================= [7654232] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 01:15:10 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654232/ [7654233] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/exports} time: 00:52:52 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654233/ [7654234] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}} time: 00:25:29 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654234/ [7654235] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:12:26 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654235/ [7654236] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:37:42 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654236/ [7654238] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} time: 00:54:48 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654238/ [7654239] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}} time: 01:04:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654239/ [7654240] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/fragment} time: 00:21:04 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654240/ [7654241] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:13:07 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654241/ [7654244] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:53:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654244/ [7654245] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:47:31 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654245/ [7654246] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/mdtest} time: 00:16:44 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654246/ [7654247] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:15:28 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654247/ [7654248] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:55:53 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654248/ [7654249] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:22:58 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654249/ [7654250] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:27 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654250/ [7654252] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/mds-flush} time: 00:09:36 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654252/ [7654253] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:12:10 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654253/ [7654254] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:40:09 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654254/ [7654255] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:19:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654255/ [7654256] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:19:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654256/ [7654257] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/mds-full} time: 00:24:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654257/ [7654258] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:23:12 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654258/ [7654259] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 01:15:21 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654259/ [7654261] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/mds_creation_retry} time: 00:14:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654261/ [7654264] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:06:21 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654264/ [7654265] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/finisher_per_module}} time: 00:09:34 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654265/ [7654267] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/metrics} time: 00:25:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654267/ [7654268] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/multimds_misc} time: 00:19:22 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654268/ [7654269] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:12:51 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654269/ [7654270] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:24:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654270/ [7654271] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/openfiletable} time: 00:14:44 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654271/ [7654272] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:21:45 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654272/ [7654273] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:41:14 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654273/ [7654274] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:18:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654274/ [7654275] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest clusters/small-cluster conf/{client mds mgr mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/trim-i24137} time: 00:11:47 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654275/ [7654277] fs/full/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-node-1-mds-1-osd conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/mgr-osd-full} time: 00:19:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654277/ [7654278] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client} time: 00:13:04 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654278/ [7654280] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:23:48 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654280/ [7654281] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests} time: 01:09:03 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654281/ [7654282] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:39:51 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654282/ [7654284] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:18:52 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654284/ [7654285] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} time: 00:49:44 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654285/ [7654286] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:53:23 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654286/ [7654287] fs/top/{begin/{0-install 1-ceph 2-logrotate 3-modules} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/fstop} time: 00:16:16 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654287/ [7654290] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} time: 00:09:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654290/ [7654291] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:29:36 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654291/ [7654292] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:45:45 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654292/ [7654295] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:28 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654295/ [7654297] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:14:23 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654297/ [7654298] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:36:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654298/ [7654299] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quota} time: 00:16:33 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654299/ [7654300] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} time: 00:13:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654300/ [7654301] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-iozone}} time: 02:24:55 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654301/ [7654302] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} time: 00:22:05 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654302/ [7654303] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/recovery-fs} time: 00:10:04 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654303/ [7654304] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:23:22 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654304/ [7654306] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/scrub} time: 00:29:58 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654306/ [7654307] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:19:52 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654307/ [7654308] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:46:03 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654308/ [7654309] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:45:40 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654309/ [7654311] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:17:34 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654311/ [7654312] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:16:25 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654312/ [7654313] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/sessionmap} time: 00:19:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654313/ [7654314] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:12:56 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654314/ [7654315] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 01:07:22 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654315/ [7654316] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} time: 02:04:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654316/ [7654317] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/ior-shared-file} time: 00:14:23 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654317/ [7654318] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:11:14 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654318/ [7654319] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:43:35 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654319/ [7654320] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:12:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654320/ [7654322] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/dbench}} time: 00:52:10 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654322/ [7654323] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap_schedule_snapdir} time: 00:12:58 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654323/ [7654324] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:17:18 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654324/ [7654325] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/snapshot}} time: 00:37:03 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654325/ [7654326] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 00:43:35 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654326/ [7654327] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:42:26 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654327/ [7654328] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snapshots} time: 00:36:32 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654328/ [7654329] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:48:23 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654329/ [7654330] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:18:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654330/ [7654331] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/strays} time: 00:27:36 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654331/ [7654332] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:10 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654332/ [7654334] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:33 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654334/ [7654335] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:18:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654335/ [7654336] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/ino_release_cb} time: 00:13:47 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654336/ [7654337] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:19:09 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654337/ [7654338] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/test_journal_migration} time: 00:20:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654338/ [7654339] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:48:06 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654339/ [7654341] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}} time: 00:19:23 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654341/ [7654342] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/truncate_delay} time: 00:13:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654342/ [7654344] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:39:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654344/ [7654345] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 01:05:04 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654345/ [7654346] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/workunit/dir-max-entries} time: 00:11:18 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654346/ [7654348] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:02 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654348/ [7654349] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:33:12 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654349/ [7654351] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/mdtest} time: 00:25:18 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654351/ [7654352] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:27:34 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654352/ [7654353] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:33:56 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654353/ [7654354] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/basic}} time: 01:42:49 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654354/ [7654355] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:13:05 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654355/ [7654356] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:38:34 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654356/ [7654357] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/workunit/quota} time: 00:14:53 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654357/ [7654358] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:21:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654358/ [7654359] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:17:02 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654359/ [7654360] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:51 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654360/ [7654362] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:12:50 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654362/ [7654364] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:40:39 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654364/ [7654365] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:24:05 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654365/ [7654367] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:46 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654367/ [7654368] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 00:52:49 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654368/ [7654369] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:29:25 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654369/ [7654370] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:13:39 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654370/ [7654371] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:19:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654371/ [7654372] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:34:09 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654372/ [7654373] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/asok_dump_tree} time: 00:13:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654373/ [7654375] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-dbench}} time: 01:33:36 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654375/ [7654376] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:22:08 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654376/ [7654377] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}} time: 01:04:31 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654377/ [7654378] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 00:52:15 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654378/ [7654379] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/auto-repair} time: 00:15:31 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654379/ [7654380] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} time: 00:23:48 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654380/ [7654381] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:13:02 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654381/ [7654382] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:31:45 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654382/ [7654383] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/backtrace} time: 00:12:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654383/ [7654384] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:16 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654384/ [7654385] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/{ignorelist_health pg_health}} time: 00:23:42 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654385/ [7654386] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs/{frag test}} time: 00:31:46 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654386/ [7654387] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests} time: 01:09:21 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654387/ [7654388] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:13:55 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654388/ [7654389] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:15:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654389/ [7654391] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 02:04:53 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654391/ [7654392] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/dbench validater/lockdep} time: 00:51:58 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654392/ [7654393] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:40 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654393/ [7654394] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/cap-flush} time: 00:10:29 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654394/ [7654395] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:16:31 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654395/ [7654396] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} time: 00:29:30 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654396/ [7654397] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} time: 00:13:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654397/ [7654398] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/client-limits} time: 00:21:33 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654398/ [7654399] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:31:34 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654399/ [7654400] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:32:22 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654400/ [7654401] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/client-readahead} time: 00:15:06 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654401/ [7654402] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:38:42 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654402/ [7654403] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:45:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654403/ [7654404] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:11:03 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654404/ [7654406] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/client-recovery} time: 00:48:40 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654406/ [7654407] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/finisher_per_module}} time: 00:14:52 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654407/ [7654409] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:49:16 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654409/ [7654410] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} time: 00:59:03 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654410/ [7654411] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:26:23 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654411/ [7654412] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:18:56 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654412/ [7654413] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:18:47 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654413/ [7654414] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:24:57 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654414/ [7654416] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/data-scan} time: 05:26:47 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654416/ [7654418] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/ior-shared-file} time: 00:15:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654418/ [7654419] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:21:01 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654419/ [7654420] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:30:17 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654420/ [7654421] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/exports} time: 00:43:55 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654421/ [7654422] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:17:47 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654422/ [7654423] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} time: 00:39:26 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654423/ [7654425] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:18:42 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654425/ [7654427] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} time: 01:13:00 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654427/ [7654428] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/fragment} time: 00:22:59 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654428/ [7654429] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:37 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654429/ [7654430] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:16:22 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654430/ [7654431] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} time: 00:16:09 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654431/ [7654433] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/journal-repair} time: 00:17:43 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654433/ [7654434] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:16:36 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654434/ [7654435] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs_python} time: 00:13:26 info: https://pulpito.ceph.com/vshankar-2024-04-12_17:34:28-fs-wip-vshankar-testing-20240412.042930-testing-default-smithi/7654435/