2024-02-17T07:00:42.797 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-17T07:00:42.798 INFO:teuthology.results:subset: '111/120000' 2024-02-17T07:00:42.798 INFO:teuthology.results:seed: '2794' 2024-02-17T16:29:25.104 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-17T16:29:25.133 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-17T23:18:01.282 INFO:teuthology.results:Tests finished! gathering results... 2024-02-17T23:18:17.819 INFO:scrape:Found 194 jobs 2024-02-17T23:18:18.186 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.186 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.186 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.186 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.203 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.204 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.204 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.204 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.216 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.216 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.216 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.216 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.430 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.431 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.431 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.431 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.443 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.443 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.443 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.443 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.453 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.453 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.453 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.453 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.611 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.611 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.611 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.612 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.625 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.625 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.625 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.625 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.635 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.636 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.636 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.636 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:18:18.912 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563375/teuthology.log 2024-02-17T23:19:23.088 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.089 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.089 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.089 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.181 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.181 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.182 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.182 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.381 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.382 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.382 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.382 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.435 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.436 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.436 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.436 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.629 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.630 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.630 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.630 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.685 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.685 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.685 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.685 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.878 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.878 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.878 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.878 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.931 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.932 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.932 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:23.932 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:24.122 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:24.122 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:24.122 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:24.123 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:24.176 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:24.177 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:24.177 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:24.177 WARNING:scrape:Saw end of BT but not start 2024-02-17T23:19:24.597 INFO:scrape:Found 21 distinct failure reasons 2024-02-17T23:19:24.598 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'} 2024-02-17T23:19:24.598 INFO:scrape:1 jobs: ['7563122'] 2024-02-17T23:19:24.598 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2024-02-17T23:19:24.598 INFO:scrape: 2024-02-17T23:19:24.598 INFO:scrape:Failure: Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6822c7b54e949006be7a0802650d01723d834d1d pull' 2024-02-17T23:19:24.599 INFO:scrape:6 jobs: ['7563211', '7563481', '7563357', '7563095', '7563080', '7563363'] 2024-02-17T23:19:24.599 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2024-02-17T23:19:24.599 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-client', '2-logrotate}', '2-scrub/no', '2-upgrade', '3-compat_client/quincy}}', '3-snaps/no', '4-flush/no', '5-workunit/kernel_untar_build}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'multimds/yes', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-octopus', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-02-17T23:19:24.599 INFO:scrape: 2024-02-17T23:19:24.599 INFO:scrape:Dead: None 2024-02-17T23:19:24.599 INFO:scrape:11 jobs: ['7563375', '7563223', '7563037', '7562987', '7563015', '7563033', '7563003', '7563025', '7563000', '7562995', '7563202'] 2024-02-17T23:19:24.600 INFO:scrape: 2024-02-17T23:19:24.600 INFO:scrape:Failure: valgrind error: Leak_StillReachable malloc malloc strdup 2024-02-17T23:19:24.600 INFO:scrape:4 jobs: ['7563008', '7563315', '7563004', '7563199'] 2024-02-17T23:19:24.600 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2024-02-17T23:19:24.600 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'tasks/fsstress', 'tasks/mirror}}', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-02-17T23:19:24.600 INFO:scrape: 2024-02-17T23:19:24.600 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6822c7b54e949006be7a0802650d01723d834d1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2024-02-17T23:19:24.600 INFO:scrape:1 jobs: ['7563005'] 2024-02-17T23:19:24.601 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2024-02-17T23:19:24.601 INFO:scrape: 2024-02-17T23:19:24.601 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-02-17T23:19:24.601 INFO:scrape:2 jobs: ['7563262', '7563062'] 2024-02-17T23:19:24.601 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'replication/default}', 'session_timeout}', 'standby-replay', 'wsync/no}'] 2024-02-17T23:19:24.601 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-02-17T23:19:24.601 INFO:scrape: 2024-02-17T23:19:24.601 INFO:scrape:Failure: Command failed on smithi134 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6822c7b54e949006be7a0802650d01723d834d1d' 2024-02-17T23:19:24.602 INFO:scrape:1 jobs: ['7563318'] 2024-02-17T23:19:24.602 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_latest}', 'fs/32bits/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{faked-ino', 'tasks/cfuse_workunit_suites_fsstress}'] 2024-02-17T23:19:24.602 INFO:scrape: 2024-02-17T23:19:24.602 INFO:scrape:Failure: Command failed on smithi001 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.39 deep-scrub' 2024-02-17T23:19:24.602 INFO:scrape:1 jobs: ['7563205'] 2024-02-17T23:19:24.602 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2024-02-17T23:19:24.603 INFO:scrape: 2024-02-17T23:19:24.603 INFO:scrape:Failure: "2024-02-17T09:11:06.654689+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log 2024-02-17T23:19:24.603 INFO:scrape:4 jobs: ['7563012', '7563414', '7563292', '7563378'] 2024-02-17T23:19:24.603 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-02-17T23:19:24.603 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/test_o_trunc}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/iogen}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2024-02-17T23:19:24.603 INFO:scrape: 2024-02-17T23:19:24.603 INFO:scrape:Failure: "1708180372.6434872 mon.smithi057 (mon.0) 683 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log 2024-02-17T23:19:24.604 INFO:scrape:5 jobs: ['7563308', '7563193', '7563011', '7563101', '7563417'] 2024-02-17T23:19:24.604 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-17T23:19:24.604 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-17T23:19:24.604 INFO:scrape: 2024-02-17T23:19:24.604 INFO:scrape:Failure: SSH connection to smithi133 was lost: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main' 2024-02-17T23:19:24.604 INFO:scrape:1 jobs: ['7563484'] 2024-02-17T23:19:24.604 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/auto-repair}'] 2024-02-17T23:19:24.604 INFO:scrape: 2024-02-17T23:19:24.605 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi059 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6822c7b54e949006be7a0802650d01723d834d1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-02-17T23:19:24.605 INFO:scrape:1 jobs: ['7563327'] 2024-02-17T23:19:24.605 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2024-02-17T23:19:24.605 INFO:scrape: 2024-02-17T23:19:24.605 INFO:scrape:Failure: Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) 2024-02-17T23:19:24.605 INFO:scrape:1 jobs: ['7563143'] 2024-02-17T23:19:24.605 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/multifs-auth}'] 2024-02-17T23:19:24.605 INFO:scrape: 2024-02-17T23:19:24.605 INFO:scrape:Failure: Command failed on smithi005 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ca0538c-cd7c-11ee-95ba-87774f69a715 -e sha1=6822c7b54e949006be7a0802650d01723d834d1d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-02-17T23:19:24.605 INFO:scrape:4 jobs: ['7563042', '7563351', '7563226', '7563471'] 2024-02-17T23:19:24.606 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-17T23:19:24.606 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-17T23:19:24.606 INFO:scrape: 2024-02-17T23:19:24.606 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi125.front.sepia.ceph.com: ['type=AVC msg=audit(1708179000.369:192): avc: denied { checkpoint_restore } for pid=1187 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-02-17T23:19:24.606 INFO:scrape:3 jobs: ['7563299', '7562985', '7563039'] 2024-02-17T23:19:24.606 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{centos_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2024-02-17T23:19:24.606 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'subvol_versions/create_subvol_version_v2', 'tasks/client-recovery}', 'tasks/snap-schedule}', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2024-02-17T23:19:24.606 INFO:scrape: 2024-02-17T23:19:24.606 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-02-17T23:19:24.606 INFO:scrape:1 jobs: ['7563250'] 2024-02-17T23:19:24.607 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-02-17T23:19:24.607 INFO:scrape: 2024-02-17T23:19:24.607 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6822c7b54e949006be7a0802650d01723d834d1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-02-17T23:19:24.607 INFO:scrape:1 jobs: ['7563187'] 2024-02-17T23:19:24.607 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2024-02-17T23:19:24.607 INFO:scrape: 2024-02-17T23:19:24.607 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-02-17T23:19:24.607 INFO:scrape:1 jobs: ['7563402'] 2024-02-17T23:19:24.607 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/xfstests-dev}'] 2024-02-17T23:19:24.608 INFO:scrape: 2024-02-17T23:19:24.608 INFO:scrape:Failure: Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) 2024-02-17T23:19:24.608 INFO:scrape:1 jobs: ['7563444'] 2024-02-17T23:19:24.608 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/admin}'] 2024-02-17T23:19:24.608 INFO:scrape: 2024-02-17T23:19:24.608 INFO:scrape:Failure: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) 2024-02-17T23:19:24.608 INFO:scrape:1 jobs: ['7563010'] 2024-02-17T23:19:24.608 INFO:scrape: 2024-02-17T23:19:24.608 INFO:scrape:Failure: No module named 'tasks.fs' 2024-02-17T23:19:24.608 INFO:scrape:1 jobs: ['7563137'] 2024-02-17T23:19:24.609 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-17T23:19:24.609 INFO:scrape: 2024-02-17T23:19:24.621 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/ failed: 40 dead: 12 running: 0 waiting: 0 queued: 0 passed: 142 Fail ================================================================= [7562985] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:41:24 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562985/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562985/ SELinux denials found on ubuntu@smithi133.front.sepia.ceph.com: ['type=AVC msg=audit(1708153751.757:192): avc: denied { checkpoint_restore } for pid=1191 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7563004] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:03:42 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563004/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563004/ valgrind error: Leak_StillReachable malloc malloc strdup [7563005] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:52:24 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563005/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563005/ sentry: https://sentry.ceph.com/organizations/ceph/?query=854cf8a8b1ab4a48b1fe954bfdc8a6df Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6822c7b54e949006be7a0802650d01723d834d1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7563008] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:49:52 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563008/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563008/ valgrind error: Leak_StillReachable malloc malloc strdup [7563010] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/backtrace} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563010/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563010/ sentry: https://sentry.ceph.com/organizations/ceph/?query=827a780aca094468815c0d892de9a20b ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) [7563011] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:07 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563011/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563011/ "1708161081.9036 mon.smithi063 (mon.0) 707 : cluster [WRN] Replacing daemon mds.cephfs.smithi063.wcqapw as rank 0 with standby daemon mds.cephfs.smithi063.rkhqcy" in cluster log [7563012] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:22:51 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563012/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563012/ "2024-02-17T09:11:06.654689+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7563039] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-recovery} ----------------------------------------------------------------- time: 00:54:26 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563039/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563039/ SELinux denials found on ubuntu@smithi164.front.sepia.ceph.com: ['type=AVC msg=audit(1708164075.049:192): avc: denied { checkpoint_restore } for pid=1189 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7563042] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:55:45 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563042/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563042/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cbeacbb8fc624f1990a84b934abd5bb0 Command failed on smithi005 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ca0538c-cd7c-11ee-95ba-87774f69a715 -e sha1=6822c7b54e949006be7a0802650d01723d834d1d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' [7563062] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:25:54 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563062/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563062/ error during scrub thrashing: rank damage found: {'backtrace'} [7563080] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:15 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563080/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563080/ Command failed on smithi006 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.39 deep-scrub' [7563095] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:08:38 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563095/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563095/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d8c429430224f729f9665aa44a5a37d Command failed on smithi088 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:6822c7b54e949006be7a0802650d01723d834d1d pull' [7563101] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:53 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563101/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563101/ "1708170075.0354486 mon.smithi040 (mon.0) 716 : cluster [WRN] Replacing daemon mds.cephfs.smithi040.lwqjwu as rank 0 with standby daemon mds.cephfs.smithi158.xxrimg" in cluster log [7563122] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:42:13 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563122/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563122/ The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'} [7563137] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:16:20 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563137/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563137/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4055101f47ad43e8936e58c5dfd7a87b No module named 'tasks.fs' [7563143] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:15:54 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563143/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563143/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eca273c69c4b4179b5d39c8c5420ef07 Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) [7563187] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:51:10 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563187/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563187/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c0730b0578a473eb855034826bbffc6 Command failed (workunit test suites/dbench.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6822c7b54e949006be7a0802650d01723d834d1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7563193] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:22 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563193/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563193/ "1708174284.7111068 mon.smithi079 (mon.0) 690 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7563199] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:44:50 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563199/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563199/ valgrind error: Leak_StillReachable malloc malloc strdup [7563205] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:32 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563205/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563205/ Command failed on smithi001 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.39 deep-scrub' [7563211] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:09:04 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563211/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563211/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f76dfcac1304faab1948a2edbc4ddbb Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:6822c7b54e949006be7a0802650d01723d834d1d pull' [7563226] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:56:31 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563226/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563226/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa3153f689a648b180284920c9ad117c Command failed on smithi094 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d08c923e-cd96-11ee-95ba-87774f69a715 -e sha1=6822c7b54e949006be7a0802650d01723d834d1d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' [7563250] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:15:02 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563250/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563250/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8f34dc37fde4593a4ec5d768236afd1 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7563262] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:32:55 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563262/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563262/ error during scrub thrashing: rank damage found: {'backtrace'} [7563292] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:26:05 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563292/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563292/ "2024-02-17T14:14:48.070031+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7563299] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} ----------------------------------------------------------------- time: 02:12:56 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563299/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563299/ SELinux denials found on ubuntu@smithi125.front.sepia.ceph.com: ['type=AVC msg=audit(1708179000.369:192): avc: denied { checkpoint_restore } for pid=1187 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7563308] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:12 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563308/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563308/ "1708180372.6434872 mon.smithi057 (mon.0) 683 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7563315] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:14:54 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563315/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563315/ valgrind error: Leak_PossiblyLost calloc __trans_list_add [7563318] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:10:23 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563318/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563318/ sentry: https://sentry.ceph.com/organizations/ceph/?query=160a9747b1154fd1837e488840f961d7 Command failed on smithi134 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6822c7b54e949006be7a0802650d01723d834d1d' [7563327] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:31:56 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563327/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=438b618f357149e28f4977e201d0b6f2 Command failed (workunit test suites/fsx.sh) on smithi059 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6822c7b54e949006be7a0802650d01723d834d1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7563351] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:56:21 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563351/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563351/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9feacea5b854819a6708080994c3e18 Command failed on smithi001 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4abb3364-cda4-11ee-95ba-87774f69a715 -e sha1=6822c7b54e949006be7a0802650d01723d834d1d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' [7563357] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:08:39 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563357/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563357/ sentry: https://sentry.ceph.com/organizations/ceph/?query=daa88153fc424385be478f17813cefe5 Command failed on smithi103 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:6822c7b54e949006be7a0802650d01723d834d1d pull' [7563363] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:54:52 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563363/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563363/ Command failed on smithi082 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.3a deep-scrub' [7563378] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:32:50 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563378/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563378/ "2024-02-17T15:17:30.348523+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7563402] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/xfstests-dev} ----------------------------------------------------------------- time: 07:53:56 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563402/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f03cab10e128426bb397cb633f6bce08 Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7563414] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:55:26 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563414/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563414/ "2024-02-17T15:39:05.303844+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7563417] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:32:40 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563417/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563417/ "1708184995.669329 mon.smithi079 (mon.0) 711 : cluster [WRN] Replacing daemon mds.cephfs.smithi079.oixgxb as rank 0 with standby daemon mds.cephfs.smithi079.oiwxrg" in cluster log [7563444] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/admin} ----------------------------------------------------------------- time: 00:35:24 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563444/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563444/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a28c484b609447198ae38e3ae440fb6 Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) [7563471] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:51:16 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563471/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=872fa0cfefdd441e8c135b1b8c634519 Command failed on smithi073 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc09c748-cdb0-11ee-95ba-87774f69a715 -e sha1=6822c7b54e949006be7a0802650d01723d834d1d -- bash -c 'ceph orch ps'" [7563481] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:31 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563481/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563481/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7d54ca33f4be4c6e9967d8a553b93bb8 Command failed on smithi047 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:6822c7b54e949006be7a0802650d01723d834d1d pull' Dead ================================================================= [7562987] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562987/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562987/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7562995] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562995/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562995/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7563000] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563000/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563000/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7563003] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563003/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563003/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7563015] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563015/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563015/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7563025] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563025/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563025/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7563033] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563033/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563033/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7563037] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563037/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563037/ Error reimaging machines: Expected smithi158's OS to be centos 9 but found ubuntu 22.04 [7563202] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563202/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563202/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7563223] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563223/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563223/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7563375] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563375/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563375/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7563484] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/auto-repair} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563484/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563484/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce9e065ae52b4f03b56d11507a97f525 SSH connection to smithi133 was lost: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main' Pass ================================================================= [7562986] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/asok_dump_tree} time: 00:19:27 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562986/ [7562988] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:04 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562988/ [7562989] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:08:48 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562989/ [7562990] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:43:59 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562990/ [7562991] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:16:53 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562991/ [7562992] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:12:39 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562992/ [7562993] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} time: 02:02:26 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562993/ [7562994] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{centos_latest} workloads/cephfs-mirror-ha-workunit} time: 00:15:13 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562994/ [7562996] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:59:26 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562996/ [7562997] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:32:06 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562997/ [7562998] fs/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} time: 00:32:59 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562998/ [7562999] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:47 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7562999/ [7563001] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:05:38 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563001/ [7563002] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} time: 00:14:49 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563002/ [7563006] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/auto-repair} time: 00:14:23 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563006/ [7563007] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:25:51 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563007/ [7563009] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:17:37 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563009/ [7563013] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/cap-flush} time: 00:13:37 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563013/ [7563014] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:18:20 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563014/ [7563017] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:19:50 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563017/ [7563019] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:11:26 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563019/ [7563021] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-limits} time: 00:23:31 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563021/ [7563023] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}} time: 00:26:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563023/ [7563027] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/client-readahead} time: 00:08:47 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563027/ [7563029] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:25:05 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563029/ [7563031] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:27 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563031/ [7563035] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:43 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563035/ [7563045] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:16:52 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563045/ [7563048] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:20:41 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563048/ [7563051] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:38 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563051/ [7563054] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:06:05 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563054/ [7563057] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:47:17 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563057/ [7563065] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} time: 00:49:58 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563065/ [7563068] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 00:40:35 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563068/ [7563071] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:40:39 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563071/ [7563074] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:48:45 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563074/ [7563077] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:15:48 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563077/ [7563083] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:35:39 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563083/ [7563086] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/exports} time: 00:49:06 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563086/ [7563089] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:11:47 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563089/ [7563092] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:13:24 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563092/ [7563098] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} time: 00:19:44 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563098/ [7563104] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:11:51 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563104/ [7563107] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:08:57 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563107/ [7563110] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:11:41 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563110/ [7563113] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:29:02 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563113/ [7563116] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/fragment} time: 00:16:08 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563116/ [7563119] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 00:46:14 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563119/ [7563125] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:22:39 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563125/ [7563128] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:03 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563128/ [7563131] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds-flush} time: 00:09:16 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563131/ [7563134] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:13:22 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563134/ [7563140] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:19:05 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563140/ [7563147] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:36:59 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563147/ [7563150] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:37:16 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563150/ [7563153] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds-full} time: 00:19:28 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563153/ [7563156] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:31:39 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563156/ [7563159] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:38:04 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563159/ [7563162] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds_creation_retry} time: 00:16:42 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563162/ [7563165] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:41 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563165/ [7563169] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:50 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563169/ [7563172] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:45:25 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563172/ [7563175] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:48 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563175/ [7563177] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:46:34 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563177/ [7563181] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:23:57 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563181/ [7563184] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/metrics} time: 00:33:47 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563184/ [7563190] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:12:32 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563190/ [7563196] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/multimds_misc} time: 00:14:28 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563196/ [7563208] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/openfiletable} time: 00:13:17 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563208/ [7563214] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} time: 01:19:49 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563214/ [7563217] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:04:25 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563217/ [7563220] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} time: 00:13:58 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563220/ [7563229] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quota} time: 00:11:35 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563229/ [7563232] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:47 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563232/ [7563235] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:18:08 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563235/ [7563238] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:20:06 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563238/ [7563241] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:31 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563241/ [7563244] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:16:00 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563244/ [7563247] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:11:38 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563247/ [7563253] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:00:57 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563253/ [7563256] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:13:22 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563256/ [7563259] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/dbench}} time: 00:50:53 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563259/ [7563265] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/recovery-fs} time: 00:11:48 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563265/ [7563268] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:17:44 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563268/ [7563271] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:25:02 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563271/ [7563274] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/scrub} time: 00:29:26 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563274/ [7563277] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:42:40 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563277/ [7563280] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:12:58 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563280/ [7563283] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:12:13 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563283/ [7563286] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:13:01 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563286/ [7563289] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/sessionmap} time: 00:11:34 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563289/ [7563295] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}} time: 00:17:03 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563295/ [7563302] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:19:33 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563302/ [7563305] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:16:22 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563305/ [7563311] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap_schedule_snapdir} time: 00:13:23 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563311/ [7563321] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:56 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563321/ [7563324] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:44:32 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563324/ [7563330] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 01:19:04 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563330/ [7563333] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snapshots} time: 00:21:01 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563333/ [7563336] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:38:32 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563336/ [7563339] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:14:05 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563339/ [7563342] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:20:32 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563342/ [7563345] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:35:12 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563345/ [7563348] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/strays} time: 00:31:41 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563348/ [7563354] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:11:37 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563354/ [7563360] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/test_journal_migration} time: 00:20:06 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563360/ [7563366] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:49:18 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563366/ [7563369] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/truncate_delay} time: 00:08:57 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563369/ [7563372] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:15:57 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563372/ [7563381] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:10:07 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563381/ [7563384] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:09:49 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563384/ [7563387] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:27:44 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563387/ [7563390] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:10:02 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563390/ [7563393] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:33 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563393/ [7563396] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:35:10 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563396/ [7563399] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/new_ops/{newops}}} time: 00:28:48 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563399/ [7563405] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:13:31 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563405/ [7563408] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:50:10 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563408/ [7563411] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/workunit/quota} time: 00:11:09 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563411/ [7563420] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:17:55 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563420/ [7563423] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 00:49:12 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563423/ [7563426] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/acls} time: 00:10:05 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563426/ [7563429] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:16:08 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563429/ [7563432] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:12:44 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563432/ [7563435] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:55:54 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563435/ [7563438] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} time: 00:57:05 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563438/ [7563441] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:09:56 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563441/ [7563447] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 01:22:27 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563447/ [7563450] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:30:49 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563450/ [7563453] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-pjd}} time: 00:18:17 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563453/ [7563456] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:34:50 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563456/ [7563459] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/alternate-pool} time: 00:13:35 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563459/ [7563462] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:10 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563462/ [7563465] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:29 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563465/ [7563468] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:12:20 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563468/ [7563475] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/asok_dump_tree} time: 00:13:37 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563475/ [7563478] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:16:46 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563478/ [7563486] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 01:01:43 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563486/ [7563488] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:53:14 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563488/ [7563490] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:46:03 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563490/ [7563492] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:10:44 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563492/ [7563494] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/backtrace} time: 00:13:32 info: https://pulpito.ceph.com/yuriw-2024-02-16_20:13:25-fs-main-distro-default-smithi/7563494/