2024-03-17T04:01:19.338 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-17T04:01:19.338 INFO:teuthology.results:subset: '186/512' 2024-03-17T04:01:19.338 INFO:teuthology.results:seed: '4120' 2024-03-17T10:32:52.670 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-17T10:32:52.694 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-17T21:40:50.994 INFO:teuthology.results:Tests finished! gathering results... 2024-03-17T21:41:12.588 INFO:scrape:Found 243 jobs 2024-03-17T21:41:20.251 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.252 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.252 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.252 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.597 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.597 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.598 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.598 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.916 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.916 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.917 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:20.917 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.227 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.228 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.228 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.228 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.550 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.551 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.551 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.551 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.861 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.861 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.861 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:21.861 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:22.182 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:22.182 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:22.182 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:22.182 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:22.504 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:22.504 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:22.504 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:22.504 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:30.213 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:30.213 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:30.214 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:30.214 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:34.664 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599125/teuthology.log 2024-03-17T21:41:35.279 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:35.279 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:35.279 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:35.279 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.803 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.803 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.803 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.803 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.879 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.880 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.880 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.880 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.962 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.962 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.962 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:57.962 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.040 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.040 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.040 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.040 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.120 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.120 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.120 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.121 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.199 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.199 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.199 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.199 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.276 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.277 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.277 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.277 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.356 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.356 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.356 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.357 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.433 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.433 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.433 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.433 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.512 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.512 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.513 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.513 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.593 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.593 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.593 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.593 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.673 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.673 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.673 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.673 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.760 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.760 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.760 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.761 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.838 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.838 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.838 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.838 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.920 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.920 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.920 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:58.920 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:59.002 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:59.002 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:59.002 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:59.002 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:59.080 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:59.080 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:59.080 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:41:59.081 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.210 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.211 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.211 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.211 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.321 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.321 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.322 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.322 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.325 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.326 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.326 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.326 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.330 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.330 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.330 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.330 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.346 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.346 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.346 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.346 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.352 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.352 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.352 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.352 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.715 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.716 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.716 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.716 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.809 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.810 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.810 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.810 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.831 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.832 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.832 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:09.832 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:11.944 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:11.945 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:11.945 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:11.945 WARNING:scrape:Saw end of BT but not start 2024-03-17T21:42:18.189 INFO:scrape:Found 19 distinct failure reasons 2024-03-17T21:42:18.189 INFO:scrape:Failure: "2024-03-17T05:45:56.474481+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log 2024-03-17T21:42:18.190 INFO:scrape:37 jobs 2024-03-17T21:42:18.191 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-03-17T21:42:18.191 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-03-17T21:42:18.191 INFO:scrape: 2024-03-17T21:42:18.191 INFO:scrape:Failure: valgrind error: Leak_StillReachable malloc malloc strdup 2024-03-17T21:42:18.191 INFO:scrape:4 jobs: ['7599286', '7599350', '7599250', '7599152'] 2024-03-17T21:42:18.191 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse'] 2024-03-17T21:42:18.192 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'tasks/mirror}}', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-03-17T21:42:18.192 INFO:scrape: 2024-03-17T21:42:18.192 INFO:scrape:Failure: "1710671146.137913 mon.smithi012 (mon.0) 679 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log 2024-03-17T21:42:18.192 INFO:scrape:4 jobs: ['7599338', '7599149', '7599300', '7599240'] 2024-03-17T21:42:18.192 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-17T21:42:18.192 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-17T21:42:18.192 INFO:scrape: 2024-03-17T21:42:18.192 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-17T21:42:18.192 INFO:scrape:2 jobs: ['7599179', '7599203'] 2024-03-17T21:42:18.193 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2024-03-17T21:42:18.193 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-03-17T21:42:18.193 INFO:scrape: 2024-03-17T21:42:18.193 INFO:scrape:Dead: b'2024-03-17T21:05:32.586 DEBUG:teuthology.exit:Finished running handlers' 2024-03-17T21:42:18.193 INFO:scrape:3 jobs: ['7599299', '7599122', '7599320'] 2024-03-17T21:42:18.193 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2024-03-17T21:42:18.193 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/yes', '3-inline/yes', '3-snaps/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/fs/misc}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fail_fs/yes', 'frag', 'fs/functional/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/1', 'roles', 'session_timeout}', 'standby-replay', 'subvol_versions/create_subvol_version_v1', 'syntax}', 'tasks/xfstests-dev}', 'tasks/{0-from/quincy', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-03-17T21:42:18.193 INFO:scrape: 2024-03-17T21:42:18.194 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi078.front.sepia.ceph.com: ['type=AVC msg=audit(1710656434.526:200): avc: denied { checkpoint_restore } for pid=1212 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-17T21:42:18.194 INFO:scrape:5 jobs: ['7599186', '7599204', '7599118', '7599256', '7599307'] 2024-03-17T21:42:18.194 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{centos_latest}', 'mds', 'mon', 'mount', 'osd}'] 2024-03-17T21:42:18.194 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/ffsb}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/3', 'session_timeout', 'session_timeout}', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'tasks/acls}', 'tasks/fsstress', 'tasks/mds-full}', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'validater/lockdep}'] 2024-03-17T21:42:18.194 INFO:scrape: 2024-03-17T21:42:18.194 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-03-17T21:42:18.194 INFO:scrape:4 jobs: ['7599334', '7599129', '7599138', '7599269'] 2024-03-17T21:42:18.194 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/random', 'session_timeout}', 'standby-replay'] 2024-03-17T21:42:18.195 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/postgres}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}', 'wsync/yes}'] 2024-03-17T21:42:18.195 INFO:scrape: 2024-03-17T21:42:18.195 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-17T21:42:18.195 INFO:scrape:1 jobs: ['7599164'] 2024-03-17T21:42:18.195 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-17T21:42:18.195 INFO:scrape: 2024-03-17T21:42:18.195 INFO:scrape:Failure: Command failed on smithi169 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 54578c2c-e430-11ee-95c9-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-17T21:42:18.195 INFO:scrape:1 jobs: ['7599223'] 2024-03-17T21:42:18.195 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-17T21:42:18.195 INFO:scrape: 2024-03-17T21:42:18.196 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi066 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-17T21:42:18.196 INFO:scrape:4 jobs: ['7599148', '7599257', '7599317', '7599207'] 2024-03-17T21:42:18.196 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-03-17T21:42:18.196 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-03-17T21:42:18.196 INFO:scrape: 2024-03-17T21:42:18.196 INFO:scrape:Failure: Command failed on smithi078 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.28 deep-scrub' 2024-03-17T21:42:18.196 INFO:scrape:3 jobs: ['7599210', '7599260', '7599333'] 2024-03-17T21:42:18.196 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2024-03-17T21:42:18.197 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2024-03-17T21:42:18.197 INFO:scrape: 2024-03-17T21:42:18.197 INFO:scrape:Dead: None 2024-03-17T21:42:18.197 INFO:scrape:1 jobs: ['7599125'] 2024-03-17T21:42:18.197 INFO:scrape: 2024-03-17T21:42:18.197 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fbdaed80d4a] 2024-03-17T21:42:18.197 INFO:scrape:ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fbdae454db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fbdae4a154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fbdaed80d4a] 6: /usr/lib64/ceph/libceph-common.so.2(+0x180eae) [0x7fbdaed80eae] 7: (MDCache::handle_cache_rejoin_ack(boost::intrusive_ptr const&)+0x2093) [0x560843c4ebb3] 8: (MDCache::dispatch(boost::intrusive_ptr const&)+0x3d4) [0x560843c6b034] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x560843b1544b] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560843b1595c] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x560843afcf25] 12: (DispatchQueue::entry()+0x53a) [0x7fbdaef7669a] 13: /usr/lib64/ceph/libceph-common.so.2(+0x408921) [0x7fbdaf008921] 14: /lib64/libc.so.6(+0x9f802) [0x7fbdae49f802] 15: /lib64/libc.so.6(+0x3f450) [0x7fbdae43f450] 2024-03-17T21:42:18.197 INFO:scrape:1 jobs: ['7599230'] 2024-03-17T21:42:18.197 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2024-03-17T21:42:18.197 INFO:scrape: 2024-03-17T21:42:18.198 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-17T21:42:18.198 INFO:scrape:3 jobs: ['7599180', '7599133', '7599265'] 2024-03-17T21:42:18.198 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-17T21:42:18.198 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-17T21:42:18.198 INFO:scrape: 2024-03-17T21:42:18.198 INFO:scrape:Failure: machine smithi179.front.sepia.ceph.com is not locked 2024-03-17T21:42:18.198 INFO:scrape:1 jobs: ['7599222'] 2024-03-17T21:42:18.198 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/permission/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_pjd}'] 2024-03-17T21:42:18.198 INFO:scrape: 2024-03-17T21:42:18.198 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi094 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-17T21:42:18.199 INFO:scrape:1 jobs: ['7599242'] 2024-03-17T21:42:18.199 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2024-03-17T21:42:18.199 INFO:scrape: 2024-03-17T21:42:18.199 INFO:scrape:Failure: SSH connection to smithi039 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2024-03-17T21:42:18.199 INFO:scrape:1 jobs: ['7599305'] 2024-03-17T21:42:18.199 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2024-03-17T21:42:18.199 INFO:scrape: 2024-03-17T21:42:18.199 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-03-17T21:42:18.199 INFO:scrape:1 jobs: ['7599306'] 2024-03-17T21:42:18.200 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-03-17T21:42:18.200 INFO:scrape: 2024-03-17T21:42:18.200 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-03-17T21:42:18.200 INFO:scrape:1 jobs: ['7599283'] 2024-03-17T21:42:18.200 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-17T21:42:18.200 INFO:scrape: 2024-03-17T21:42:18.214 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/ failed: 74 dead: 4 running: 0 waiting: 0 queued: 0 passed: 165 Fail ================================================================= [7599114] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:36:41 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599114/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599114/ "2024-03-17T04:25:14.262700+0000 mon.a (mon.0) 194 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599118] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:41:07 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599118/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599118/ SELinux denials found on ubuntu@smithi142.front.sepia.ceph.com: ['type=AVC msg=audit(1710649163.373:199): avc: denied { checkpoint_restore } for pid=1203 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7599124] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:34:03 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599124/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599124/ "2024-03-17T04:42:03.977179+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599129] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:43:30 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599129/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599129/ The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} [7599131] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:35:18 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599131/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599131/ "2024-03-17T04:55:27.749882+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599133] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:28 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599133/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599133/ sentry: https://sentry.ceph.com/organizations/ceph/?query=47201d969aa04c27a4fdc39d1a4c39a5 Command failed (workunit test suites/fsstress.sh) on smithi130 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7599134] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:05:45 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599134/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599134/ "2024-03-17T05:00:43.704118+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599138] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:58:45 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599138/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599138/ The following counters failed to be set on mds daemons: {'mds_server.req_rmsnap_latency.avgcount'} [7599139] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:28:30 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599139/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599139/ "2024-03-17T05:15:48.615562+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599145] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:33:06 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599145/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599145/ "2024-03-17T05:18:43.565681+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599148] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:28:08 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599148/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599148/ sentry: https://sentry.ceph.com/organizations/ceph/?query=078b2f4be67f48d784ac078da5c16284 Command failed (workunit test suites/fsx.sh) on smithi066 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7599149] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:13 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599149/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599149/ "1710653590.002462 mon.smithi111 (mon.0) 689 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7599152] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:48:06 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599152/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599152/ valgrind error: Leak_StillReachable malloc malloc strdup [7599153] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:32:25 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599153/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599153/ "2024-03-17T05:36:57.578733+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599157] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:36:21 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599157/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599157/ "2024-03-17T05:45:56.474481+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599164] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:12:24 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599164/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599164/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9970d7626f0446a99a4f51170e71b367 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7599166] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:57:43 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599166/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599166/ "2024-03-17T05:59:46.636207+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599170] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:28:31 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599170/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599170/ "2024-03-17T06:16:51.587472+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599175] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:30:58 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599175/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599175/ "2024-03-17T06:26:49.749057+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599179] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:48:48 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599179/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599179/ error during scrub thrashing: rank damage found: {'backtrace'} [7599180] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:35:59 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599180/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599180/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2947f2f392f4d6db4ea007438e2e54f Command failed (workunit test suites/fsstress.sh) on smithi154 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7599185] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:39:46 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599185/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599185/ "2024-03-17T06:32:14.382228+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599186] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds-full} ----------------------------------------------------------------- time: 00:23:32 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599186/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599186/ SELinux denials found on ubuntu@smithi078.front.sepia.ceph.com: ['type=AVC msg=audit(1710656434.526:200): avc: denied { checkpoint_restore } for pid=1212 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7599189] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:31:54 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599189/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599189/ "2024-03-17T06:40:41.631362+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599194] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:36:23 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599194/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599194/ "2024-03-17T06:51:34.456772+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599203] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:31:18 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599203/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599203/ error during scrub thrashing: rank damage found: {'backtrace'} [7599204] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:14:22 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599204/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599204/ SELinux denials found on ubuntu@smithi170.front.sepia.ceph.com: ['type=AVC msg=audit(1710658687.837:197): avc: denied { checkpoint_restore } for pid=1162 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7599207] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:32:03 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599207/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599207/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd3640eb7efc4422a6200641b808feee Command failed (workunit test suites/fsx.sh) on smithi017 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7599209] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:28:36 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599209/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599209/ "2024-03-17T07:19:24.099584+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599210] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:20:36 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599210/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599210/ Command failed on smithi078 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.28 deep-scrub' [7599212] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:33:48 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599212/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599212/ "2024-03-17T07:24:44.362126+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599215] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:38:27 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599215/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599215/ "2024-03-17T07:28:15.787243+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599222] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599222/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599222/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5646b0b3be9a4383b4dc2e5aba7fca3b machine smithi179.front.sepia.ceph.com is not locked [7599223] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:56:18 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599223/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599223/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b5d1f80cbaf49be945ba6927aad12db Command failed on smithi169 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 54578c2c-e430-11ee-95c9-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' [7599228] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:34:30 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599228/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599228/ "2024-03-17T07:45:46.061560+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599230] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:28:48 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599230/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599230/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aee8f8ab34e64d3e8c59baf526a81c8b Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [7599233] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:41:08 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599233/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599233/ "2024-03-17T07:50:21.431337+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599240] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:31 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599240/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599240/ "1710662391.5787323 mon.smithi152 (mon.0) 716 : cluster [WRN] Replacing daemon mds.cephfs.smithi152.vsdvdu as rank 0 with standby daemon mds.cephfs.smithi176.ebzhcf" in cluster log [7599242] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:59:13 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599242/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599242/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35a505721ed041fe92d945d68a653922 Command failed (workunit test suites/dbench.sh) on smithi094 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7599246] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:53:01 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599246/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599246/ "2024-03-17T08:03:08.367072+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599250] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:49:35 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599250/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599250/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7599252] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:34:52 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599252/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599252/ "2024-03-17T08:16:21.605134+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599256] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:22:48 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599256/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599256/ SELinux denials found on ubuntu@smithi178.front.sepia.ceph.com: ['type=AVC msg=audit(1710662915.976:200): avc: denied { checkpoint_restore } for pid=1205 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7599257] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:38:09 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599257/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599257/ sentry: https://sentry.ceph.com/organizations/ceph/?query=afbc7baac8d44b858a9af2ac63daa3da Command failed (workunit test suites/fsx.sh) on smithi039 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7599259] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:28:50 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599259/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599259/ "2024-03-17T08:23:29.849636+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599260] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:22:14 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599260/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599260/ Command failed on smithi059 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.30 deep-scrub' [7599262] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:35:44 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599262/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599262/ "2024-03-17T08:24:42.091852+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599264] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 01:08:09 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599264/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599264/ "2024-03-17T08:27:17.539098+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599265] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:07 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599265/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599265/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9bcd357e25794748955a3fc8fa0bf063 Command failed (workunit test suites/fsstress.sh) on smithi107 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7599267] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:35:01 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599267/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599267/ "2024-03-17T08:35:28.500883+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599269] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:24:44 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599269/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599269/ The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'} [7599283] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:35:56 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599283/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599283/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50bba2bbf86b4181966cb905b33575dd Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7599286] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:02:38 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599286/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599286/ valgrind error: Leak_StillReachable malloc malloc strdup [7599288] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:34:58 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599288/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599288/ "2024-03-17T09:06:40.355615+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599292] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:57:35 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599292/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599292/ "2024-03-17T09:04:40.618197+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599295] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:45:26 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599295/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599295/ "2024-03-17T09:13:39.538007+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599297] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:13:27 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599297/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599297/ "2024-03-17T09:14:17.163314+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599300] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:17 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599300/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599300/ "1710667561.815724 mon.smithi174 (mon.0) 685 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7599301] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:49:53 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599301/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599301/ "2024-03-17T09:24:56.750579+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599305] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:15:52 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599305/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599305/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6094645025b5459b97b70f03d66bf809 SSH connection to smithi039 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [7599306] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:41:33 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599306/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599306/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e6dff9a09284d76b273fe70ab2c0198 Command failed (workunit test suites/ffsb.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7599307] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/acls} ----------------------------------------------------------------- time: 00:18:26 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599307/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599307/ SELinux denials found on ubuntu@smithi046.front.sepia.ceph.com: ['type=AVC msg=audit(1710666648.446:199): avc: denied { checkpoint_restore } for pid=1270 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7599310] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:36:16 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599310/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599310/ "2024-03-17T09:38:21.295765+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599317] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:24:12 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599317/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599317/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a4f89e9ea8743458c2ed7c5fc6fe0f7 Command failed (workunit test suites/fsx.sh) on smithi094 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bb04bdb70687c7e8c6d06f0a15c8daac247883 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7599318] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:35:18 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599318/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599318/ "2024-03-17T09:53:33.350027+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599327] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:38:22 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599327/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599327/ "2024-03-17T10:06:24.278783+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599333] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:51:01 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599333/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599333/ Command failed on smithi001 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.39 deep-scrub' [7599334] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:36:21 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599334/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599334/ The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} [7599338] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:31:43 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599338/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599338/ "1710671146.137913 mon.smithi012 (mon.0) 679 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7599339] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:48:42 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599339/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599339/ "2024-03-17T10:23:13.909273+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599345] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:18:01 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599345/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599345/ "2024-03-17T10:31:11.212778+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599348] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:42:54 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599348/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599348/ "2024-03-17T10:39:16.996607+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7599350] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:45:59 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599350/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599350/ valgrind error: Leak_StillReachable malloc malloc strdup [7599355] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:03:20 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599355/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599355/ "2024-03-17T10:51:15.503020+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log Dead ================================================================= [7599122] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599122/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599122/ hit max job timeout [7599125] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-limits} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599125/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599125/ Error reimaging machines: Failed to power on smithi044 [7599299] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599299/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599299/ hit max job timeout [7599320] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599320/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599320/ hit max job timeout Pass ================================================================= [7599115] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:16:52 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599115/ [7599116] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:15:43 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599116/ [7599117] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:14:31 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599117/ [7599119] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:39:54 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599119/ [7599120] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:31:59 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599120/ [7599121] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/cap-flush} time: 00:09:26 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599121/ [7599123] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:20 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599123/ [7599126] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:55 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599126/ [7599127] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:12:23 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599127/ [7599128] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:25 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599128/ [7599130] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:13:12 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599130/ [7599132] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/client-readahead} time: 00:11:32 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599132/ [7599135] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:11:43 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599135/ [7599136] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/dbench}} time: 00:51:02 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599136/ [7599137] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-recovery} time: 00:57:40 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599137/ [7599140] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:11:28 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599140/ [7599141] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:20:03 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599141/ [7599142] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:19:54 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599142/ [7599143] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:48:27 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599143/ [7599144] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:16:32 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599144/ [7599146] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} time: 01:18:47 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599146/ [7599147] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:02:38 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599147/ [7599150] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:13:04 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599150/ [7599151] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 00:50:11 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599151/ [7599154] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}} time: 00:41:12 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599154/ [7599155] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/exports} time: 00:46:12 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599155/ [7599156] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:43:10 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599156/ [7599158] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:32 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599158/ [7599159] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:17:38 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599159/ [7599160] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:18:31 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599160/ [7599161] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:44 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599161/ [7599162] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:33:22 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599162/ [7599163] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:10:35 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599163/ [7599165] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:27:38 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599165/ [7599167] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:39:50 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599167/ [7599168] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} time: 00:18:21 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599168/ [7599169] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:44:40 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599169/ [7599171] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:32:25 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599171/ [7599172] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:24:50 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599172/ [7599173] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/fragment} time: 00:20:06 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599173/ [7599174] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:47:17 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599174/ [7599176] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:16:57 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599176/ [7599177] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:42:25 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599177/ [7599178] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:12:04 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599178/ [7599181] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds-flush} time: 00:09:24 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599181/ [7599182] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:13:04 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599182/ [7599183] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 01:00:42 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599183/ [7599184] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:15:05 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599184/ [7599187] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:12:54 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599187/ [7599188] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:57 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599188/ [7599190] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:20 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599190/ [7599191] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:40:16 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599191/ [7599192] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:21:51 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599192/ [7599193] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:41 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599193/ [7599195] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds_creation_retry} time: 00:09:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599195/ [7599196] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:33:07 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599196/ [7599197] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 00:55:01 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599197/ [7599198] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:48 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599198/ [7599199] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:25 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599199/ [7599200] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:08:35 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599200/ [7599201] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:34:55 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599201/ [7599202] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/metrics} time: 00:27:11 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599202/ [7599205] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:11:53 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599205/ [7599206] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/new_ops/{newops}}} time: 00:11:08 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599206/ [7599208] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/multimds_misc} time: 00:19:37 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599208/ [7599211] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:21:45 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599211/ [7599213] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/openfiletable} time: 00:10:41 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599213/ [7599214] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:16:21 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599214/ [7599216] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} time: 00:22:46 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599216/ [7599217] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:11:02 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599217/ [7599218] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} time: 00:09:41 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599218/ [7599219] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:31:40 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599219/ [7599220] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:10:40 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599220/ [7599221] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:09:46 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599221/ [7599224] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:16:12 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599224/ [7599225] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quota} time: 00:14:33 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599225/ [7599226] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:25:12 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599226/ [7599227] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:46 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599227/ [7599229] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/recovery-fs} time: 00:13:01 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599229/ [7599231] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 02:28:41 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599231/ [7599232] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/scrub} time: 00:23:44 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599232/ [7599234] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:41:26 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599234/ [7599235] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:49:30 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599235/ [7599236] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:10:54 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599236/ [7599237] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:19:09 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599237/ [7599238] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:25:51 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599238/ [7599239] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:58:04 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599239/ [7599241] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/sessionmap} time: 00:15:48 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599241/ [7599243] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:00:47 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599243/ [7599244] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-pjd}} time: 00:18:30 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599244/ [7599245] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:34:44 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599245/ [7599247] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} time: 02:05:17 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599247/ [7599248] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:12:44 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599248/ [7599249] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:03 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599249/ [7599251] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:14:59 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599251/ [7599253] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:41 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599253/ [7599254] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:25:50 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599254/ [7599255] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap_schedule_snapdir} time: 00:09:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599255/ [7599258] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snapshots} time: 00:26:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599258/ [7599261] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:06 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599261/ [7599263] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/strays} time: 00:30:37 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599263/ [7599266] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:12:50 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599266/ [7599268] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/test_journal_migration} time: 00:14:46 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599268/ [7599270] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:15:49 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599270/ [7599271] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/truncate_delay} time: 00:13:27 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599271/ [7599272] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:53:38 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599272/ [7599273] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:09:05 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599273/ [7599274] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:44:17 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599274/ [7599275] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:14:56 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599275/ [7599276] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:13:21 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599276/ [7599277] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{centos_latest} tasks/mirror} time: 01:55:39 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599277/ [7599278] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{centos_latest} workloads/cephfs-mirror-ha-workunit} time: 00:16:19 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599278/ [7599279] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:59:32 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599279/ [7599280] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:32:24 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599280/ [7599281] fs/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} time: 00:32:59 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599281/ [7599282] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:20 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599282/ [7599284] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:41:16 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599284/ [7599285] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/fstop} time: 00:10:41 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599285/ [7599287] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:45:14 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599287/ [7599289] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:56 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599289/ [7599290] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:44:58 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599290/ [7599291] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:51:12 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599291/ [7599293] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 00:43:17 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599293/ [7599294] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/workunit/dir-max-entries} time: 00:11:06 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599294/ [7599296] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:15:51 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599296/ [7599298] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:12:23 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599298/ [7599302] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/workunit/quota} time: 00:13:52 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599302/ [7599303] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:28:12 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599303/ [7599304] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:16:28 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599304/ [7599308] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:00:34 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599308/ [7599309] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} time: 00:25:26 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599309/ [7599311] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/admin} time: 02:40:07 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599311/ [7599312] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:57 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599312/ [7599313] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:54 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599313/ [7599314] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:08:52 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599314/ [7599315] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:36 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599315/ [7599316] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:50:48 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599316/ [7599319] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/alternate-pool} time: 00:09:34 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599319/ [7599321] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:16:21 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599321/ [7599322] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:33:12 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599322/ [7599323] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:36 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599323/ [7599324] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:10:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599324/ [7599325] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:49:41 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599325/ [7599326] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/asok_dump_tree} time: 00:18:00 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599326/ [7599328] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:16:05 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599328/ [7599329] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:25:25 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599329/ [7599330] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:42:16 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599330/ [7599331] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:13:09 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599331/ [7599332] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/auto-repair} time: 00:14:10 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599332/ [7599335] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:12:26 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599335/ [7599336] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:32:34 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599336/ [7599337] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/backtrace} time: 00:09:23 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599337/ [7599340] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:20 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599340/ [7599341] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:12:20 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599341/ [7599342] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:08:48 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599342/ [7599343] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:14 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599343/ [7599344] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/cap-flush} time: 00:13:32 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599344/ [7599346] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-ffsb}} time: 02:23:35 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599346/ [7599347] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:13:10 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599347/ [7599349] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:24:18 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599349/ [7599351] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-limits} time: 00:31:18 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599351/ [7599352] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 01:05:14 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599352/ [7599353] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:37:38 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599353/ [7599354] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:13:56 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599354/ [7599356] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/client-readahead} time: 00:09:01 info: https://pulpito.ceph.com/teuthology-2024-03-13_21:24:03-fs-squid-distro-default-smithi/7599356/