2024-03-13T05:46:12.033 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-13T05:46:12.033 INFO:teuthology.results:subset: '832/1024' 2024-03-13T05:46:12.033 INFO:teuthology.results:seed: '6045' 2024-03-13T06:25:17.311 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-13T06:25:17.344 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-13T14:29:56.229 INFO:teuthology.results:Tests finished! gathering results... 2024-03-13T14:30:01.194 INFO:scrape:Found 58 jobs 2024-03-13T14:30:03.130 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.131 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.131 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.131 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.134 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.134 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.135 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.135 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.378 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.378 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.378 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.378 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.382 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.382 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.382 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.382 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.628 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.628 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.628 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.629 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.632 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.632 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.632 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.632 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.905 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.906 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.906 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.906 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.910 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.910 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.910 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.910 WARNING:scrape:Saw end of BT but not start 2024-03-13T14:30:03.981 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596362/teuthology.log 2024-03-13T14:30:19.402 INFO:scrape:Found 13 distinct failure reasons 2024-03-13T14:30:19.403 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0ea271bb96f7f4cec2f2a40ff88a5def1e0530f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2024-03-13T14:30:19.403 INFO:scrape:1 jobs: ['7596350'] 2024-03-13T14:30:19.403 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2024-03-13T14:30:19.403 INFO:scrape: 2024-03-13T14:30:19.404 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-13T14:30:19.404 INFO:scrape:3 jobs: ['7596377', '7596378', '7596327'] 2024-03-13T14:30:19.404 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/automatic', 'session_timeout}', 'standby-replay'] 2024-03-13T14:30:19.404 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-03-13T14:30:19.404 INFO:scrape: 2024-03-13T14:30:19.404 INFO:scrape:Dead: None 2024-03-13T14:30:19.404 INFO:scrape:3 jobs: ['7596362', '7596360', '7596344'] 2024-03-13T14:30:19.404 INFO:scrape: 2024-03-13T14:30:19.404 INFO:scrape:Failure: "2024-03-13T06:40:00.001081+0000 mon.smithi087 (mon.0) 530 : cluster 3 [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid" in cluster log 2024-03-13T14:30:19.405 INFO:scrape:1 jobs: ['7596346'] 2024-03-13T14:30:19.405 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-13T14:30:19.405 INFO:scrape: 2024-03-13T14:30:19.405 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-13T14:30:19.405 INFO:scrape:5 jobs: ['7596358', '7596342', '7596375', '7596367', '7596352'] 2024-03-13T14:30:19.405 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-03-13T14:30:19.406 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-03-13T14:30:19.406 INFO:scrape: 2024-03-13T14:30:19.406 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a87fde6e633694345631fcca2fd8a5f190eef61c 2024-03-13T14:30:19.406 INFO:scrape:7 jobs: ['7596355', '7596349', '7596379', '7596324', '7596374', '7596369', '7596328'] 2024-03-13T14:30:19.406 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}'] 2024-03-13T14:30:19.406 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/ffsb}}', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/ubuntu_latest', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'frag', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/shell/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{ignorelist_health', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'subvol_versions/create_subvol_version_v2', 'tasks/cephfs-shell}', 'tasks/cephfs_misc_tests}', 'tasks/ior-shared-file}', 'tasks/pool-perm}', 'tasks/volumes/{overrides', 'tasks/{1-thrash/mon', 'test/basic}}', 'thrashosds-health}'] 2024-03-13T14:30:19.406 INFO:scrape: 2024-03-13T14:30:19.406 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0ea271bb96f7f4cec2f2a40ff88a5def1e0530f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-03-13T14:30:19.407 INFO:scrape:1 jobs: ['7596338'] 2024-03-13T14:30:19.407 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-03-13T14:30:19.407 INFO:scrape: 2024-03-13T14:30:19.407 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi143 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0ea271bb96f7f4cec2f2a40ff88a5def1e0530f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none pjd' 2024-03-13T14:30:19.407 INFO:scrape:1 jobs: ['7596364'] 2024-03-13T14:30:19.407 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-pjd}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2024-03-13T14:30:19.407 INFO:scrape: 2024-03-13T14:30:19.407 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-03-13T14:30:19.407 INFO:scrape:1 jobs: ['7596347'] 2024-03-13T14:30:19.407 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-03-13T14:30:19.407 INFO:scrape: 2024-03-13T14:30:19.408 INFO:scrape:Failure: valgrind error: Leak_StillReachable malloc malloc strdup 2024-03-13T14:30:19.408 INFO:scrape:2 jobs: ['7596359', '7596373'] 2024-03-13T14:30:19.408 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'validater/valgrind}'] 2024-03-13T14:30:19.408 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-03-13T14:30:19.408 INFO:scrape: 2024-03-13T14:30:19.408 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-03-13T14:30:19.408 INFO:scrape:1 jobs: ['7596370'] 2024-03-13T14:30:19.408 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-13T14:30:19.408 INFO:scrape: 2024-03-13T14:30:19.408 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi110 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0ea271bb96f7f4cec2f2a40ff88a5def1e0530f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-13T14:30:19.409 INFO:scrape:1 jobs: ['7596341'] 2024-03-13T14:30:19.409 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-03-13T14:30:19.409 INFO:scrape: 2024-03-13T14:30:19.409 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) 2024-03-13T14:30:19.409 INFO:scrape:1 jobs: ['7596372'] 2024-03-13T14:30:19.409 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2024-03-13T14:30:19.409 INFO:scrape: 2024-03-13T14:30:19.413 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/ failed: 25 dead: 3 running: 0 waiting: 0 queued: 0 passed: 30 Fail ================================================================= [7596324] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596324/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596324/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de636e295e8748b8905573179ddfcf62 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sh a1=a87fde6e633694345631fcca2fd8a5f190eef61c [7596327] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:17:23 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596327/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596327/ error during scrub thrashing: rank damage found: {'backtrace'} [7596328] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596328/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1d4a716205f45faa89f86e7f5f996ce Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sh a1=a87fde6e633694345631fcca2fd8a5f190eef61c [7596338] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:40:32 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596338/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596338/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b048f512f234a69b3d2a1881a98c750 Command failed (workunit test suites/ffsb.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0ea271bb96f7f4cec2f2a40ff88a5def1e0530f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7596341] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:23:35 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596341/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596341/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3848f6593fe4437eaa0909875cae71bc Command failed (workunit test suites/fsx.sh) on smithi110 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0ea271bb96f7f4cec2f2a40ff88a5def1e0530f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7596342] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:48:17 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596342/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596342/ reached maximum tries (51) after waiting for 300 seconds [7596346] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:42:15 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596346/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596346/ "2024-03-13T06:40:00.001081+0000 mon.smithi087 (mon.0) 530 : cluster 3 [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid" in cluster log [7596347] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 08:23:40 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596347/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596347/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d65174cc925849639c157ab3960be3fd Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7596349] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596349/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a6df8e168264319a141f787cd715bca Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sh a1=a87fde6e633694345631fcca2fd8a5f190eef61c [7596350] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:35:24 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596350/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596350/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c6fd3fe033e46a7823b1c8deacd1d25 Command failed (workunit test suites/pjd.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0ea271bb96f7f4cec2f2a40ff88a5def1e0530f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7596352] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:45:17 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596352/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596352/ reached maximum tries (51) after waiting for 300 seconds [7596355] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596355/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596355/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6de2cf0f1c68406b938f49b818426777 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sh a1=a87fde6e633694345631fcca2fd8a5f190eef61c [7596358] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:44:57 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596358/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596358/ reached maximum tries (51) after waiting for 300 seconds [7596359] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:50:45 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596359/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596359/ valgrind error: Leak_StillReachable malloc malloc strdup [7596364] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-pjd}} ----------------------------------------------------------------- time: 00:14:13 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596364/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596364/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a66ace330be44929a2eec12b9e7d5463 Command failed (workunit test fs/fscrypt.sh) on smithi143 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0ea271bb96f7f4cec2f2a40ff88a5def1e0530f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none pjd' [7596367] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:44:58 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596367/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596367/ reached maximum tries (51) after waiting for 300 seconds [7596369] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596369/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596369/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03a3000024ce4fd782286cfd68174424 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sh a1=a87fde6e633694345631fcca2fd8a5f190eef61c [7596370] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:44:45 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596370/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596370/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c30a53df6b4944ecaa5124f5a6766a48 Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7596372] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:32:50 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596372/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e98444630fe5443face865a060d0f10c Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) [7596373] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:13:30 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596373/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596373/ valgrind error: Leak_PossiblyLost calloc __trans_list_add [7596374] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596374/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596374/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4398397337745829b62c8efe8308022 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sh a1=a87fde6e633694345631fcca2fd8a5f190eef61c [7596375] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:31 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596375/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596375/ reached maximum tries (51) after waiting for 300 seconds [7596377] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:31:56 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596377/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596377/ error during scrub thrashing: rank damage found: {'backtrace'} [7596378] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:01:54 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596378/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596378/ error during scrub thrashing: rank damage found: {'backtrace'} [7596379] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/pool-perm} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596379/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596379/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29c12056b76f4540884fdfbde9e0315a Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sh a1=a87fde6e633694345631fcca2fd8a5f190eef61c Dead ================================================================= [7596344] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596344/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596344/ Error reimaging machines: Failed to power on smithi189 [7596360] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596360/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596360/ Error reimaging machines: Failed to power on smithi083 [7596362] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/exports} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596362/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596362/ Error reimaging machines: Failed to power on smithi139 Pass ================================================================= [7596323] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:19:46 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596323/ [7596325] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:52:14 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596325/ [7596326] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/openfiletable} time: 00:14:29 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596326/ [7596329] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:37:10 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596329/ [7596330] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 00:38:07 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596330/ [7596331] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:15:21 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596331/ [7596332] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:14:02 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596332/ [7596333] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:24:29 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596333/ [7596334] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/quota} time: 00:15:37 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596334/ [7596335] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:13:59 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596335/ [7596336] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:06:19 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596336/ [7596337] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:53:47 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596337/ [7596339] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:36:53 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596339/ [7596340] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap_schedule_snapdir} time: 00:09:48 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596340/ [7596343] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:12 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596343/ [7596345] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:15:18 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596345/ [7596348] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:24:15 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596348/ [7596351] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/acls} time: 00:15:35 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596351/ [7596353] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:33:06 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596353/ [7596354] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:00:03 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596354/ [7596356] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:22:42 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596356/ [7596357] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:57:33 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596357/ [7596361] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:51:03 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596361/ [7596363] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:56 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596363/ [7596365] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:37:36 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596365/ [7596366] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/fragment} time: 00:16:04 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596366/ [7596368] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:34:48 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596368/ [7596371] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 00:56:04 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596371/ [7596376] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:48:39 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596376/ [7596380] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:06 info: https://pulpito.ceph.com/vshankar-2024-03-13_05:41:30-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7596380/