2023-09-26T14:10:00.233 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-26T14:10:00.233 INFO:teuthology.results:subset: '830/1024' 2023-09-26T14:10:00.233 INFO:teuthology.results:seed: '3911' 2023-09-27T10:28:54.547 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-27T10:28:54.577 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-27T21:48:48.076 INFO:teuthology.results:Tests finished! gathering results... 2023-09-27T21:49:05.902 INFO:scrape:Found 202 jobs 2023-09-27T21:49:06.137 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.137 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.137 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.137 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.388 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.388 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.388 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.388 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.607 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.607 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.607 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:06.607 WARNING:scrape:Saw end of BT but not start 2023-09-27T21:49:12.386 WARNING:scrape:Misunderstood line: 2023-09-27T21:49:12.472 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402428/teuthology.log 2023-09-27T21:49:15.517 WARNING:scrape:Misunderstood line: 2023-09-27T21:49:18.335 INFO:scrape:Found 13 distinct failure reasons 2023-09-27T21:49:18.336 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-09-27T21:49:18.336 INFO:scrape:1 jobs: ['7402493'] 2023-09-27T21:49:18.336 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2023-09-27T21:49:18.336 INFO:scrape: 2023-09-27T21:49:18.336 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2023-09-27T21:49:18.336 INFO:scrape:1 jobs: ['7402514'] 2023-09-27T21:49:18.336 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-09-27T21:49:18.337 INFO:scrape: 2023-09-27T21:49:18.337 INFO:scrape:Failure: "2023-09-26T20:12:49.999242+0000 mon.a (mon.0) 8047 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-09-27T21:49:18.337 INFO:scrape:4 jobs: ['7402430', '7402501', '7402330', '7402491'] 2023-09-27T21:49:18.337 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-09-27T21:49:18.337 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/failover}', 'tasks/scrub}', 'tasks/snap-schedule}'] 2023-09-27T21:49:18.337 INFO:scrape: 2023-09-27T21:49:18.338 INFO:scrape:Failure: Command failed on smithi071 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 419a543c-5c97-11ee-8db4-212e2dc638e7 -- bash -c 'ceph fs set cephfs allow_standby_replay true'" 2023-09-27T21:49:18.338 INFO:scrape:1 jobs: ['7402395'] 2023-09-27T21:49:18.338 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-09-27T21:49:18.338 INFO:scrape: 2023-09-27T21:49:18.338 INFO:scrape:Failure: timeout expired in wait_until_healthy 2023-09-27T21:49:18.338 INFO:scrape:3 jobs: ['7402405', '7402421', '7402427'] 2023-09-27T21:49:18.338 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-09-27T21:49:18.339 INFO:scrape:suites union: ['1-client', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-upgrade', '3-compat_client/no}}', '3-compat_client/quincy}}', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8', 'centos_8.stream_container_tools', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fail_fs/no', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg-warn}', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-octopus'] 2023-09-27T21:49:18.339 INFO:scrape: 2023-09-27T21:49:18.339 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-09-27T21:49:18.339 INFO:scrape:1 jobs: ['7402480'] 2023-09-27T21:49:18.339 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}'] 2023-09-27T21:49:18.339 INFO:scrape: 2023-09-27T21:49:18.339 INFO:scrape:Dead: b'2023-09-27T02:16:15.370 DEBUG:teuthology.exit:Finished running handlers' 2023-09-27T21:49:18.339 INFO:scrape:6 jobs: ['7402327', '7402458', '7402361', '7402509', '7402411', '7402380'] 2023-09-27T21:49:18.340 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-09-27T21:49:18.340 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/no', '3-inline/no', '3-inline/yes', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-verify}', '5-upgrade-with-workload', '5-workunit/kernel_untar_build}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fail_fs/yes', 'frag', 'fs/functional/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/xfstests-dev}', 'tasks/{0-from/quincy', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2023-09-27T21:49:18.340 INFO:scrape: 2023-09-27T21:49:18.340 INFO:scrape:Failure: Command failed on smithi006 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-09-27T21:49:18.340 INFO:scrape:1 jobs: ['7402466'] 2023-09-27T21:49:18.340 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-09-27T21:49:18.341 INFO:scrape: 2023-09-27T21:49:18.341 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-09-27T21:49:18.341 INFO:scrape:1 jobs: ['7402482'] 2023-09-27T21:49:18.341 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-09-27T21:49:18.341 INFO:scrape: 2023-09-27T21:49:18.341 INFO:scrape:Failure: SSH connection to smithi099 was lost: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main' 2023-09-27T21:49:18.341 INFO:scrape:2 jobs: ['7402429', '7402436'] 2023-09-27T21:49:18.341 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-09-27T21:49:18.342 INFO:scrape:suites union: ['1-ceph', '1-client', '2-logrotate}', '2-upgrade', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'centos_8', 'clusters/1-mds-2-client', 'clusters/1-mds-2-client-micro', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'multimds/yes', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/mdtest}', 'tasks/{0-octopus'] 2023-09-27T21:49:18.342 INFO:scrape: 2023-09-27T21:49:18.342 INFO:scrape:Dead: None 2023-09-27T21:49:18.342 INFO:scrape:2 jobs: ['7402428', '7402432'] 2023-09-27T21:49:18.342 INFO:scrape: 2023-09-27T21:49:18.342 INFO:scrape:Failure: Test failure: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) 2023-09-27T21:49:18.342 INFO:scrape:1 jobs: ['7402404'] 2023-09-27T21:49:18.342 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2023-09-27T21:49:18.343 INFO:scrape: 2023-09-27T21:49:18.343 INFO:scrape:Failure: Test failure: test_cephfs_mirror_incremental_sync (tasks.cephfs.test_mirroring.TestMirroring) 2023-09-27T21:49:18.343 INFO:scrape:1 jobs: ['7402473'] 2023-09-27T21:49:18.343 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2023-09-27T21:49:18.343 INFO:scrape: 2023-09-27T21:49:18.352 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/ failed: 15 dead: 10 running: 0 waiting: 0 queued: 0 passed: 177 Fail ================================================================= [7402330] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 01:52:56 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402330/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402330/ "2023-09-26T14:49:34.334517+0000 mon.a (mon.0) 654 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7402395] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:56:37 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402395/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=195acb21b57e4ac3b7e58bfd080df5c3 Command failed on smithi071 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 419a543c-5c97-11ee-8db4-212e2dc638e7 -- bash -c 'ceph fs set cephfs allow_standby_replay true'" [7402404] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:51:20 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402404/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ebb2607c5a742958abfc7815bfbd800 Test failure: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) [7402405] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 01:03:34 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402405/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402405/ sentry: https://sentry.ceph.com/organizations/ceph/?query=69d69abad44a4fe28be1c61db8c241d6 timeout expired in wait_until_healthy [7402421] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:30:13 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402421/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402421/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e4530cab7964f46aa1e23b9c149c192 timeout expired in wait_until_healthy [7402427] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:26:46 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402427/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402427/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49ab6f479a8044cea0bcc923e01fda10 timeout expired in wait_until_healthy [7402430] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 01:16:43 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402430/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402430/ "2023-09-26T20:12:49.999242+0000 mon.a (mon.0) 8047 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7402466] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 01:00:09 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402466/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402466/ Command failed on smithi006 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' [7402473] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:03:57 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402473/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402473/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65c0009f38f042bd901ddc7c281cbada Test failure: test_cephfs_mirror_incremental_sync (tasks.cephfs.test_mirroring.TestMirroring) [7402480] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:17:57 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402480/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402480/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78012540de0243d99d1865e92af1e621 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7402482] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:28:08 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402482/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402482/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e1ea08a861f452a81131afb0bb11b85 saw valgrind issues [7402491] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:29:26 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402491/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402491/ "2023-09-27T00:30:14.248231+0000 mon.a (mon.0) 568 : cluster [WRN] Health check failed: Degraded data redundancy: 22/2328 objects degraded (0.945%), 1 pg degraded (PG_DEGRADED)" in cluster log [7402493] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:48:55 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402493/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402493/ error during scrub thrashing: rank damage found: {'backtrace'} [7402501] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 01:50:56 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402501/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402501/ "2023-09-27T01:06:21.352979+0000 mon.a (mon.0) 348 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY)" in cluster log [7402514] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:31:51 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402514/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402514/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d77c8e5b72e244b8826bfa35155f7ed9 Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) Dead ================================================================= [7402327] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402327/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402327/ hit max job timeout [7402361] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402361/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402361/ hit max job timeout [7402380] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402380/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402380/ hit max job timeout [7402411] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402411/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402411/ hit max job timeout [7402428] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402428/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402428/ Error reimaging machines: Expected smithi099's OS to be rhel 8.6 but found ubuntu 22.04 [7402429] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402429/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402429/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94d2a21817ca44aba5f4a75a5a8c50a4 SSH connection to smithi099 was lost: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main' [7402432] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402432/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402432/ Error reimaging machines: Expected smithi099's OS to be rhel 8.6 but found centos 8 [7402436] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402436/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402436/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5973e20d0b9541659fa7a709eb4212cc SSH connection to smithi099 was lost: 'sudo yum upgrade -y linux-firmware' [7402458] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402458/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402458/ hit max job timeout [7402509] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402509/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402509/ hit max job timeout Pass ================================================================= [7402326] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402326/ [7402328] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:58:26 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402328/ [7402329] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:32:59 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402329/ [7402331] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:54:29 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402331/ [7402332] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:51:46 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402332/ [7402333] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:32:52 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402333/ [7402334] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:28:04 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402334/ [7402335] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:42:48 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402335/ [7402336] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:54:25 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402336/ [7402337] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:50 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402337/ [7402338] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:15:07 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402338/ [7402339] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:19:11 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402339/ [7402340] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:26 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402340/ [7402341] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 00:47:29 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402341/ [7402342] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:28:20 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402342/ [7402343] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:18:29 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402343/ [7402344] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:28 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402344/ [7402345] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:13 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402345/ [7402346] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:33:58 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402346/ [7402347] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:46:56 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402347/ [7402348] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:19:40 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402348/ [7402349] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:43:21 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402349/ [7402350] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:19:39 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402350/ [7402351] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:17 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402351/ [7402352] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:16:17 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402352/ [7402353] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:58 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402353/ [7402354] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:18:33 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402354/ [7402355] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:19:42 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402355/ [7402356] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:22:30 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402356/ [7402357] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} time: 01:13:05 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402357/ [7402358] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:13:22 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402358/ [7402359] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:06:41 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402359/ [7402360] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:58:14 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402360/ [7402362] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:18:21 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402362/ [7402363] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:28 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402363/ [7402364] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:24:32 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402364/ [7402365] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:28:35 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402365/ [7402366] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:57 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402366/ [7402367] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:22:59 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402367/ [7402368] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:32 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402368/ [7402369] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:17:01 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402369/ [7402370] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:16:53 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402370/ [7402371] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:17:43 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402371/ [7402372] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:29:25 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402372/ [7402373] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:46:55 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402373/ [7402374] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:53 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402374/ [7402375] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 00:48:16 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402375/ [7402376] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:18:45 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402376/ [7402377] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 00:55:11 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402377/ [7402378] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:16:53 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402378/ [7402379] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:48:25 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402379/ [7402381] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:02:21 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402381/ [7402382] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:14:44 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402382/ [7402383] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:31:33 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402383/ [7402384] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:26:26 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402384/ [7402385] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:22:09 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402385/ [7402386] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:15:20 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402386/ [7402387] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 01:07:19 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402387/ [7402388] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:20:49 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402388/ [7402389] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:17:17 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402389/ [7402390] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 01:17:24 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402390/ [7402391] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:32 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402391/ [7402392] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:19:28 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402392/ [7402393] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:42:48 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402393/ [7402394] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:13:57 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402394/ [7402396] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:36 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402396/ [7402397] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 01:34:28 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402397/ [7402398] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:15:57 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402398/ [7402399] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:16:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402399/ [7402400] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:26:45 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402400/ [7402401] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:38:32 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402401/ [7402402] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:17:37 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402402/ [7402403] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:35:02 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402403/ [7402406] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:58:59 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402406/ [7402407] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:22:26 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402407/ [7402408] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:54 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402408/ [7402409] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:14:04 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402409/ [7402410] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:19:49 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402410/ [7402412] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:57:47 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402412/ [7402413] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:22:18 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402413/ [7402414] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:34:54 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402414/ [7402415] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:32 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402415/ [7402416] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:16:34 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402416/ [7402417] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:13 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402417/ [7402418] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:28:07 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402418/ [7402419] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:22:22 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402419/ [7402420] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:30:53 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402420/ [7402422] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:46:56 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402422/ [7402423] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:59:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402423/ [7402424] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:40:13 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402424/ [7402425] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:59:16 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402425/ [7402426] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:20:08 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402426/ [7402431] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:09:02 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402431/ [7402433] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:48:20 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402433/ [7402434] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:21:39 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402434/ [7402435] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:23:31 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402435/ [7402437] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:04:11 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402437/ [7402438] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-pjd}} time: 00:19:58 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402438/ [7402439] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:39:01 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402439/ [7402440] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:22:43 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402440/ [7402441] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:48 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402441/ [7402442] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:19:29 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402442/ [7402443] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:45:35 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402443/ [7402444] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:27:33 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402444/ [7402445] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 00:51:05 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402445/ [7402446] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:33:16 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402446/ [7402447] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:19:37 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402447/ [7402448] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:40 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402448/ [7402449] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:24:36 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402449/ [7402450] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:46:37 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402450/ [7402451] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:17:00 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402451/ [7402452] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:34:29 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402452/ [7402453] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:23:31 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402453/ [7402454] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:18 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402454/ [7402455] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:36:27 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402455/ [7402456] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:12 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402456/ [7402457] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:20:18 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402457/ [7402459] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:33 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402459/ [7402460] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:15:35 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402460/ [7402461] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:51:45 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402461/ [7402462] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:25:59 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402462/ [7402463] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:16:55 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402463/ [7402464] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{ubuntu_20.04} tasks/mirror} time: 02:13:45 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402464/ [7402465] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:26:25 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402465/ [7402467] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:22:27 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402467/ [7402468] fs/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/nfs} time: 00:40:01 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402468/ [7402469] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:21:58 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402469/ [7402470] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:49:35 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402470/ [7402471] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:17:34 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402471/ [7402472] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/fstop} time: 00:17:10 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402472/ [7402474] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:19:48 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402474/ [7402475] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:30:36 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402475/ [7402476] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:20:20 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402476/ [7402477] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:18:19 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402477/ [7402478] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:53:19 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402478/ [7402479] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:20:30 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402479/ [7402481] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:04 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402481/ [7402483] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 02:21:49 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402483/ [7402484] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:16:25 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402484/ [7402485] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:22:23 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402485/ [7402486] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:17:15 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402486/ [7402487] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:33:28 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402487/ [7402488] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:49:19 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402488/ [7402489] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:17:24 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402489/ [7402490] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:21 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402490/ [7402492] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:26 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402492/ [7402494] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:17:36 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402494/ [7402495] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:38 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402495/ [7402496] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:29 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402496/ [7402497] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:25:44 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402497/ [7402498] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:17:37 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402498/ [7402499] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:59:31 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402499/ [7402500] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:33:47 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402500/ [7402502] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:40:01 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402502/ [7402503] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:16:21 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402503/ [7402504] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:01 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402504/ [7402505] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:47:07 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402505/ [7402506] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:53:04 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402506/ [7402507] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:22:11 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402507/ [7402508] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:16:41 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402508/ [7402510] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:26:51 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402510/ [7402511] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:22:01 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402511/ [7402512] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:40:30 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402512/ [7402513] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:24:36 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402513/ [7402515] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:54:14 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402515/ [7402516] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:25:43 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402516/ [7402517] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:21:08 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402517/ [7402518] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:33 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402518/ [7402519] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:15:49 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402519/ [7402520] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:18:20 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402520/ [7402521] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:18:23 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402521/ [7402522] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:49:09 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402522/ [7402523] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:37:12 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402523/ [7402524] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:14:37 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402524/ [7402525] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-ffsb}} time: 01:52:32 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402525/ [7402526] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:16:25 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402526/ [7402527] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} time: 01:05:02 info: http://pulpito.ceph.com/vshankar-2023-09-26_14:09:41-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7402527/