2024-04-14T04:57:46.339 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-14T04:57:46.339 INFO:teuthology.results:subset: '22/512' 2024-04-14T04:57:46.339 INFO:teuthology.results:seed: '1489' 2024-04-14T10:23:37.662 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-14T10:23:37.689 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-14T22:02:42.295 INFO:teuthology.results:Tests finished! gathering results... 2024-04-14T22:03:03.932 INFO:scrape:Found 241 jobs 2024-04-14T22:03:19.414 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638474/teuthology.log 2024-04-14T22:03:45.132 INFO:scrape:Found 20 distinct failure reasons 2024-04-14T22:03:45.133 INFO:scrape:Failure: valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions 2024-04-14T22:03:45.133 INFO:scrape:2 jobs: ['7638372', '7638470'] 2024-04-14T22:03:45.133 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-04-14T22:03:45.133 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/1', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-04-14T22:03:45.133 INFO:scrape: 2024-04-14T22:03:45.134 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-14T22:03:45.134 INFO:scrape:2 jobs: ['7638448', '7638537'] 2024-04-14T22:03:45.134 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-14T22:03:45.134 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-14T22:03:45.134 INFO:scrape: 2024-04-14T22:03:45.134 INFO:scrape:Failure: Test failure: test_quiesce_authpin_wait (tasks.cephfs.test_quiesce.TestQuiesceMultiRank) 2024-04-14T22:03:45.134 INFO:scrape:1 jobs: ['7638426'] 2024-04-14T22:03:45.134 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/quiesce}'] 2024-04-14T22:03:45.134 INFO:scrape: 2024-04-14T22:03:45.135 INFO:scrape:Failure: Command failed on smithi078 with status 5: 'sudo systemctl stop ceph-818fcfec-fa38-11ee-bc8e-c7b262605968@mon.a' 2024-04-14T22:03:45.135 INFO:scrape:68 jobs 2024-04-14T22:03:45.136 INFO:scrape:suites intersection: [] 2024-04-14T22:03:45.136 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-mount', '1-rename', '2-logrotate', '2-scrub/no', '2-scrub/yes', '2-workload/dbench', '3-modules}', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/single-container-host', 'export-check', 'frag', 'fs/cephadm/multivolume/{0-start', 'fs/cephadm/renamevolume/{0-start', 'fs/nfs/{cluster/{1-node}', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'pg_health}}', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'supported-random-distros$/{centos_latest}', 'tasks/nfs}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-04-14T22:03:45.136 INFO:scrape: 2024-04-14T22:03:45.136 INFO:scrape:Failure: Fuse mount failed to populate/sys/ after 31 seconds 2024-04-14T22:03:45.136 INFO:scrape:1 jobs: ['7638558'] 2024-04-14T22:03:45.137 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2024-04-14T22:03:45.137 INFO:scrape: 2024-04-14T22:03:45.137 INFO:scrape:Failure: SSH connection to smithi096 was lost: 'sudo yum install -y kernel' 2024-04-14T22:03:45.137 INFO:scrape:3 jobs: ['7638443', '7638422', '7638475'] 2024-04-14T22:03:45.137 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg_health}'] 2024-04-14T22:03:45.137 INFO:scrape:suites union: ['1-ceph', '1-mount/mount/fuse', '1-upgrade}}', '2-clients/fuse-upgrade', '2-logrotate', '3-modules}', '3-workload/stress_tests/kernel_untar_build}}', 'branch/nautilus', 'centos_8.stream', 'clusters/1a3s-mds-2c-client', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro/{centos_latest}', 'fs/multifs/{begin/{0-install', 'fs/upgrade/nofs/{bluestore-bitmap', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon-debug', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no-mds-cluster', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/multifs-auth}', 'tasks/{0-install', 'tasks/{0-pacific'] 2024-04-14T22:03:45.137 INFO:scrape: 2024-04-14T22:03:45.137 INFO:scrape:Failure: "2024-04-14T08:39:04.994014+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log 2024-04-14T22:03:45.138 INFO:scrape:1 jobs: ['7638369'] 2024-04-14T22:03:45.138 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/forward-scrub}'] 2024-04-14T22:03:45.138 INFO:scrape: 2024-04-14T22:03:45.138 INFO:scrape:Failure: Command failed on smithi151 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1df6d63a-fa4a-11ee-bc8e-c7b262605968 -e sha1=d7e176b41636f150fe82aaedea82def3e9af0203 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' 2024-04-14T22:03:45.138 INFO:scrape:7 jobs: ['7638559', '7638412', '7638370', '7638386', '7638424', '7638519', '7638480'] 2024-04-14T22:03:45.138 INFO:scrape:suites intersection: ['1-volume/{0-create', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}'] 2024-04-14T22:03:45.138 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-04-14T22:03:45.138 INFO:scrape: 2024-04-14T22:03:45.138 INFO:scrape:Dead: b'2024-04-14T21:58:54.237 DEBUG:teuthology.exit:Finished running handlers' 2024-04-14T22:03:45.139 INFO:scrape:1 jobs: ['7638515'] 2024-04-14T22:03:45.139 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}'] 2024-04-14T22:03:45.139 INFO:scrape: 2024-04-14T22:03:45.139 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) 2024-04-14T22:03:45.139 INFO:scrape:1 jobs: ['7638347'] 2024-04-14T22:03:45.139 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/mirror}}'] 2024-04-14T22:03:45.139 INFO:scrape: 2024-04-14T22:03:45.139 INFO:scrape:Dead: None 2024-04-14T22:03:45.139 INFO:scrape:3 jobs: ['7638474', '7638442', '7638504'] 2024-04-14T22:03:45.139 INFO:scrape: 2024-04-14T22:03:45.140 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-04-14T22:03:45.140 INFO:scrape:1 jobs: ['7638465'] 2024-04-14T22:03:45.140 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cephfs-shell}'] 2024-04-14T22:03:45.140 INFO:scrape: 2024-04-14T22:03:45.140 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi121 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b3cf8ca51f5d8edaa7cbce710f0de3a66b6f6be9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2024-04-14T22:03:45.140 INFO:scrape:1 jobs: ['7638498'] 2024-04-14T22:03:45.140 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-14T22:03:45.140 INFO:scrape: 2024-04-14T22:03:45.140 INFO:scrape:Failure: Command failed on smithi008 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.11 deep-scrub' 2024-04-14T22:03:45.140 INFO:scrape:2 jobs: ['7638436', '7638553'] 2024-04-14T22:03:45.141 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-octopus'] 2024-04-14T22:03:45.141 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-octopus'] 2024-04-14T22:03:45.141 INFO:scrape: 2024-04-14T22:03:45.141 INFO:scrape:Failure: [Errno None] Unable to connect to port 22 on 172.21.15.29 2024-04-14T22:03:45.141 INFO:scrape:1 jobs: ['7638505'] 2024-04-14T22:03:45.141 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/no', '5-workunit/direct_io}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2024-04-14T22:03:45.141 INFO:scrape: 2024-04-14T22:03:45.141 INFO:scrape:Failure: Failed to reconnect to smithi195 2024-04-14T22:03:45.141 INFO:scrape:1 jobs: ['7638425'] 2024-04-14T22:03:45.141 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/no', '4-flush/no', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'pg_health', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}'] 2024-04-14T22:03:45.142 INFO:scrape: 2024-04-14T22:03:45.142 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2024-04-14T22:03:45.142 INFO:scrape:1 jobs: ['7638500'] 2024-04-14T22:03:45.142 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate', '3-modules}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-client'] 2024-04-14T22:03:45.142 INFO:scrape: 2024-04-14T22:03:45.142 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-04-14T22:03:45.142 INFO:scrape:1 jobs: ['7638496'] 2024-04-14T22:03:45.142 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-04-14T22:03:45.142 INFO:scrape: 2024-04-14T22:03:45.142 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-04-14T22:03:45.142 INFO:scrape:1 jobs: ['7638344'] 2024-04-14T22:03:45.143 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/cephfs-shell}'] 2024-04-14T22:03:45.143 INFO:scrape: 2024-04-14T22:03:45.143 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2024-04-14T22:03:45.143 INFO:scrape:1 jobs: ['7638509'] 2024-04-14T22:03:45.143 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'pg_health}', 'subvol_versions/create_subvol_version_v2', 'tasks/acls}'] 2024-04-14T22:03:45.143 INFO:scrape: 2024-04-14T22:03:45.157 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/ failed: 93 dead: 7 running: 0 waiting: 0 queued: 0 passed: 141 Fail ================================================================= [7638329] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:09:40 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638329/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638329/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b2a42ef2381e4ea89e66aaac763c422d Command failed on smithi060 with status 5: 'sudo systemctl stop ceph-4235db9e-fa1e-11ee-bc8e-c7b262605968@mon.a' [7638331] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:02 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638331/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638331/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b89ebe9f355e4ff9be958f2d8af394f5 Command failed on smithi102 with status 5: 'sudo systemctl stop ceph-48837308-fa1e-11ee-bc8e-c7b262605968@mon.a' [7638335] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host overrides/{ignorelist_health pg_health}} ----------------------------------------------------------------- time: 00:09:04 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638335/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638335/ sentry: https://sentry.ceph.com/organizations/ceph/?query=96888d6751e44d21a6b4ed51559d70ab Command failed on smithi046 with status 5: 'sudo systemctl stop ceph- dc0da23e-fa36-11ee-bc8e-c7b262605968@mon.smithi046' [7638342] fs/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:08:25 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638342/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=526fd8d3a1f7433e98fa532b6670ad3c Command failed on smithi162 with status 5: 'sudo systemctl stop ceph- bec173ee-fa37-11ee-bc8e-c7b262605968@mon.a' [7638344] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:37:05 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638344/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638344/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ebbc42b79b244909ad463f2733035de Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7638347] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/{ignorelist_health pg_health} tasks/mirror}} ----------------------------------------------------------------- time: 00:28:50 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638347/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638347/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bdc9b5be07b49ddb6a9c9655c4b7ff4 Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring) [7638349] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:08:17 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638349/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc29ce878f424ac381b46b4e4da4aaf5 Command failed on smithi053 with status 5: 'sudo systemctl stop ceph-c8e05016-fa37-11ee-bc8e-c7b262605968@mon.a' [7638351] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:10:56 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638351/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638351/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d017c35aedc94fd3a68cc4c90056c256 Command failed on smithi032 with status 5: 'sudo systemctl stop ceph-e55f9e20-f8d1-11ee-bc8c-c7b262605968@mon.a' [7638354] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:08:28 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638354/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638354/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0d08e253f3244c681bfc80615ab45b0 Command failed on smithi007 with status 5: 'sudo systemctl stop ceph-f4965c3c-fa37-11ee-bc8e-c7b262605968@mon.a' [7638356] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:08:58 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638356/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51c8b9cf690e450c890f7327cc555935 Command failed on smithi018 with status 5: 'sudo systemctl stop ceph-7bdcc26c-fa38-11ee-bc8e-c7b262605968@mon.a' [7638360] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:08:51 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638360/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638360/ sentry: https://sentry.ceph.com/organizations/ceph/?query=36f672cb1c054817b9f5aae61fdde97a Command failed on smithi078 with status 5: 'sudo systemctl stop ceph-818fcfec-fa38-11ee-bc8e-c7b262605968@mon.a' [7638361] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:10:42 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638361/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638361/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af92fe5816f64bda83b298c9beed87cc Command failed on smithi032 with status 5: 'sudo systemctl stop ceph-2e552fd8-fa39-11ee-bc8e-c7b262605968@mon.a' [7638366] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:09:32 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638366/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638366/ sentry: https://sentry.ceph.com/organizations/ceph/?query=486b973f42ee46be862db407298b9052 Command failed on smithi005 with status 5: 'sudo systemctl stop ceph-5a417a3e-fa39-11ee-bc8e-c7b262605968@mon.a' [7638367] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:08:34 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638367/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638367/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28242f4a40b94cb8b72559a4e2892a38 Command failed on smithi033 with status 5: 'sudo systemctl stop ceph-677f318c-fa39-11ee-bc8e-c7b262605968@mon.a' [7638369] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} ----------------------------------------------------------------- time: 00:18:55 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638369/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638369/ "2024-04-14T08:39:04.994014+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log [7638370] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:20:59 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638370/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638370/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ddbcc2b33a10495cb808b04bdb66dcec Command failed on smithi006 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 915bde74-fa39-11ee-bc8e-c7b262605968 -e sha1=d7e176b41636f150fe82aaedea82def3e9af0203 -- bash -c 'ceph orch ps'" [7638371] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:09:50 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638371/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638371/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a690b33645c4ba384e9d777afa04a9e Command failed on smithi081 with status 5: 'sudo systemctl stop ceph- ab1fcce4-fa39-11ee-bc8e-c7b262605968@mon.a' [7638372] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:56:58 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638372/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638372/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7638378] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:10:09 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638378/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638378/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c715a51ac7754708be011dbd7de56f05 Command failed on smithi064 with status 5: 'sudo systemctl stop ceph-2fca35d2-fa3b-11ee-bc8e-c7b262605968@mon.a' [7638379] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:09:37 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638379/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638379/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c41bae7198ba4b28a4132b94a80c9bf8 Command failed on smithi163 with status 5: 'sudo systemctl stop ceph-825b7b80-fa3b-11ee-bc8e-c7b262605968@mon.a' [7638382] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:08:36 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638382/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638382/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae4a056f79524f97a1cdd7d91b09bdda Command failed on smithi144 with status 5: 'sudo systemctl stop ceph-5f5735de-fa3b-11ee-bc8e-c7b262605968@mon.a' [7638386] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:20:06 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638386/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638386/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f245c3b1c4fe4812b81a6bd5c0436e5e Command failed on smithi031 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a92f287e-fa3b-11ee-bc8e-c7b262605968 -e sha1=d7e176b41636f150fe82aaedea82def3e9af0203 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' [7638387] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:08:14 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638387/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638387/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b71ca651d9d4f8587155059e8a086e0 Command failed on smithi063 with status 5: 'sudo systemctl stop ceph-d5f0c07a-fa3b-11ee-bc8e-c7b262605968@mon.a' [7638390] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:11:15 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638390/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=864096d9fd6542eca321ec6953c2f07e Command failed on smithi018 with status 5: 'sudo systemctl stop ceph-4d08c7c0-fa3c-11ee-bc8e-c7b262605968@mon.a' [7638392] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:39 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638392/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638392/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c7c23234a8442cca650948b12b9f6d7 Command failed on smithi045 with status 5: 'sudo systemctl stop ceph-131efa66-fa3c-11ee-bc8e-c7b262605968@mon.a' [7638394] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:09:47 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638394/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638394/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed9faf4300fd447ab680349d030af358 Command failed on smithi032 with status 5: 'sudo systemctl stop ceph-5202c3b6-fa3c-11ee-bc8e-c7b262605968@mon.a' [7638397] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:09:16 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638397/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d83e378d8bf14187a65d2b3266a799f8 Command failed on smithi017 with status 5: 'sudo systemctl stop ceph-4c216164-fa3c-11ee-bc8e-c7b262605968@mon.a' [7638402] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:10:11 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638402/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e9cdb5c614243b38ac8fc5152644c7c Command failed on smithi100 with status 5: 'sudo systemctl stop ceph-0be0eac4-fa3d-11ee-bc8e-c7b262605968@mon.a' [7638406] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:09:01 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638406/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638406/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de02c88c8ff14a4c94f6a02b5cf34904 Command failed on smithi022 with status 5: 'sudo systemctl stop ceph-9dc0f560-fa3d-11ee-bc8e-c7b262605968@mon.a' [7638411] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:08:02 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638411/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638411/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3068bd85b6db4d078c602edfd596216c Command failed on smithi120 with status 5: 'sudo systemctl stop ceph-f1e125d4-fa3d-11ee-bc8e-c7b262605968@mon.a' [7638412] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:23:03 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638412/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638412/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c4dda15a802451483b866c2e307ed63 Command failed on smithi012 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58ae00ac- fa3e-11ee-bc8e-c7b262605968 -e sha1=d7e176b41636f150fe82aaedea82def3e9af0203 -- bash -c 'ceph orch ps'" [7638413] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:08:43 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638413/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f23c77097144ee9aa2a0a47674696ec Command failed on smithi115 with status 5: 'sudo systemctl stop ceph-25d13e4c-fa3e-11ee-bc8e-c7b262605968@mon.a' [7638416] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:10:14 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638416/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638416/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d66a50438c2c4a239b12640191a1b95b Command failed on smithi064 with status 5: 'sudo systemctl stop ceph-a72c02ec-fa3e-11ee-bc8e-c7b262605968@mon.a' [7638419] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:08:37 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638419/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638419/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3651929c4cd4f8dae6939dbad66657e Command failed on smithi006 with status 5: 'sudo systemctl stop ceph-6dc56d22-fa3e-11ee-bc8e-c7b262605968@mon.a' [7638424] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:21:50 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638424/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638424/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3ef4f9c220b43ce934434236461e15a Command failed on smithi060 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3cb620a-fa3e-11ee-bc8e-c7b262605968 -e sha1=d7e176b41636f150fe82aaedea82def3e9af0203 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' [7638425] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638425/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638425/ sentry: https://sentry.ceph.com/organizations/ceph/?query=832bd13e891a4514959198463b2207ae Failed to reconnect to smithi195 [7638426] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/quiesce} ----------------------------------------------------------------- time: 00:36:01 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638426/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=780d5d6df2c747ad97d40f7fd78644e6 Test failure: test_quiesce_authpin_wait (tasks.cephfs.test_quiesce.TestQuiesceMultiRank) [7638428] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:09:23 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638428/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638428/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd1bc1861fd2445199fe9bc7805d9e12 Command failed on smithi003 with status 5: 'sudo systemctl stop ceph-d93f2976-fa3e-11ee-bc8e-c7b262605968@mon.a' [7638430] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:10:46 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638430/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638430/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6006dc1ac3084fc4b7376969a1a6a09f Command failed on smithi026 with status 5: 'sudo systemctl stop ceph-d77fe16a-fa3f-11ee-bc8e-c7b262605968@mon.a' [7638435] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:08:26 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638435/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638435/ sentry: https://sentry.ceph.com/organizations/ceph/?query=532bed467c80489ab468fcc51b70e1d7 Command failed on smithi007 with status 5: 'sudo systemctl stop ceph- fa6cdf5c-fa3f-11ee-bc8e-c7b262605968@mon.a' [7638436] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:07 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638436/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638436/ Command failed on smithi008 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.11 deep-scrub' [7638438] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:07:56 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638438/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638438/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa338b7e470a47198341bdfbc30fcd96 Command failed on smithi086 with status 5: 'sudo systemctl stop ceph-36ff9a04-fa40-11ee-bc8e-c7b262605968@mon.a' [7638441] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:08:43 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638441/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638441/ sentry: https://sentry.ceph.com/organizations/ceph/?query=036ef960ab8845ed9486a11609bd6428 Command failed on smithi094 with status 5: 'sudo systemctl stop ceph-88f10064-fa40-11ee-bc8e-c7b262605968@mon.a' [7638444] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:11:01 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638444/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638444/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dca93fc274224c4dafe56ae61e0b973a Command failed on smithi018 with status 5: 'sudo systemctl stop ceph-f37d17b0-fa40-11ee-bc8e-c7b262605968@mon.a' [7638446] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:08:58 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638446/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638446/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ffd5ece7f6d468c8c60d231e7899a22 Command failed on smithi017 with status 5: 'sudo systemctl stop ceph-a3bedce0-fa40-11ee-bc8e-c7b262605968@mon.a' [7638448] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:37:32 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638448/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638448/ sentry: https://sentry.ceph.com/organizations/ceph/?query=749f936038be4ebc98776fdc1d8036ed Command failed (workunit test suites/fsstress.sh) on smithi100 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b3cf8ca51f5d8edaa7cbce710f0de3a66b6f6be9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7638451] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:08:06 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638451/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638451/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c108abbdb3d480daae23c9939599dcc Command failed on smithi092 with status 5: 'sudo systemctl stop ceph- dda5243c-fa40-11ee-bc8e-c7b262605968@mon.a' [7638453] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:09:25 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638453/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638453/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e48c597c389f434eb5c67acd035ccf5f Command failed on smithi027 with status 5: 'sudo systemctl stop ceph-7e6cc3f2-fa41-11ee-bc8e-c7b262605968@mon.a' [7638456] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:11:01 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638456/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638456/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c33bdd9907d14e5d9c22092b55a74f60 Command failed on smithi069 with status 5: 'sudo systemctl stop ceph-0dc6e230-fa42-11ee-bc8e-c7b262605968@mon.a' [7638460] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/{ignorelist_health pg_health}} ----------------------------------------------------------------- time: 00:08:38 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638460/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d56d17bd3f2d45c19b831e257c928c90 Command failed on smithi176 with status 5: 'sudo systemctl stop ceph-08bfdc60-fa42-11ee-bc8e-c7b262605968@mon.smithi176' [7638465] fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:12:51 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638465/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=273921997eca44f3a57f4a45ed48e6bf Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7638467] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:09:21 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638467/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638467/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ece7e5dab9a14fd6b51d4be3cbdfe15a Command failed on smithi064 with status 5: 'sudo systemctl stop ceph-b3259f78-fa42-11ee-bc8e-c7b262605968@mon.a' [7638470] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:51:42 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638470/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638470/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7638471] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:09:08 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638471/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b3994e726ca4d589bf37838b7ed3798 Command failed on smithi006 with status 5: 'sudo systemctl stop ceph- af12d40a-fa42-11ee-bc8e-c7b262605968@mon.a' [7638478] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:09:58 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638478/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638478/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a3062e03e544744804b3f8aa6e3b6a1 Command failed on smithi106 with status 5: 'sudo systemctl stop ceph-17380230-fa43-11ee-bc8e-c7b262605968@mon.a' [7638479] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:32 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638479/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638479/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a05cc70ccea04002a7f2c6c83b2f7fcc Command failed on smithi050 with status 5: 'sudo systemctl stop ceph-b2561460-fa42-11ee-bc8e-c7b262605968@mon.a' [7638480] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:21:34 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638480/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638480/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b461e49ee11b4d3a86f0bdff7f12a26c Command failed on smithi104 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d9a10d2c-fa42-11ee-bc8e-c7b262605968 -e sha1=d7e176b41636f150fe82aaedea82def3e9af0203 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' [7638483] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:08:09 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638483/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638483/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c52f67ce0b4e4ae7994678148f3cbb3f Command failed on smithi112 with status 5: 'sudo systemctl stop ceph-76227f5a-fa43-11ee-bc8e-c7b262605968@mon.a' [7638484] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:08:39 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638484/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638484/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c0b55942ac0249b48b58a645987b689d Command failed on smithi016 with status 5: 'sudo systemctl stop ceph-e7cd76e6-fa43-11ee-bc8e-c7b262605968@mon.a' [7638487] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:10:10 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638487/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638487/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d1534d47d1f4b639dffd6f1249a52d1 Command failed on smithi032 with status 5: 'sudo systemctl stop ceph-768be4ee-fa44-11ee-bc8e-c7b262605968@mon.a' [7638489] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:08:28 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638489/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638489/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f7c824b02284137a4c8fbbf0ffa097b Command failed on smithi031 with status 5: 'sudo systemctl stop ceph-5af05ac6-fa44-11ee-bc8e-c7b262605968@mon.a' [7638493] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:08:17 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638493/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638493/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68e6e99b632a4b10a4c2db901de944d9 Command failed on smithi027 with status 5: 'sudo systemctl stop ceph- ce39fca8-fa44-11ee-bc8e-c7b262605968@mon.a' [7638496] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:13:18 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638496/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638496/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcd8060ef93c4c19916547b8b9598c8f Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7638497] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:31 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638497/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638497/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7552b63f4e344221a6e7b8c807a693f4 Command failed on smithi017 with status 5: 'sudo systemctl stop ceph-c8648a64-fa44-11ee-bc8e-c7b262605968@mon.a' [7638498] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:37:58 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638498/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638498/ sentry: https://sentry.ceph.com/organizations/ceph/?query=25611ad1103247e08bfc0a75c84ff10c Command failed (workunit test suites/fsstress.sh) on smithi121 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b3cf8ca51f5d8edaa7cbce710f0de3a66b6f6be9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7638500] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:11:32 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638500/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=847042656b744d279f6b2b6b4fbdd8ea Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7638505] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638505/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638505/ sentry: https://sentry.ceph.com/organizations/ceph/?query=089a839fd5714ac8a18d5407a67e8779 [Errno None] Unable to connect to port 22 on 172.21.15.29 [7638507] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:08:24 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638507/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638507/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1edf44517304953b4060b9f90f53820 Command failed on smithi049 with status 5: 'sudo systemctl stop ceph-1a63055c-fa45-11ee-bc8e-c7b262605968@mon.a' [7638509] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/acls} ----------------------------------------------------------------- time: 00:12:38 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638509/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638509/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd0d1c3404634add85e647eca7c45f00 Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) [7638510] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:08:34 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638510/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638510/ sentry: https://sentry.ceph.com/organizations/ceph/?query=573e41bf0ecb49e5b819da72ca63783f Command failed on smithi029 with status 5: 'sudo systemctl stop ceph-c69ca0da-fa45-11ee-bc8e-c7b262605968@mon.a' [7638511] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:08:59 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638511/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638511/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b94a5a8caae407fa6906ca290cbd427 Command failed on smithi072 with status 5: 'sudo systemctl stop ceph-b7a82f04-fa45-11ee-bc8e-c7b262605968@mon.a' [7638516] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:11:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638516/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638516/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a97021ccc77499f845e6a7c353d7454 Command failed on smithi044 with status 5: 'sudo systemctl stop ceph-43bd0c8e-fa47-11ee-bc8e-c7b262605968@mon.a' [7638518] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:09:21 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638518/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638518/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1341195dc9646efa448dcae1aeef5ac Command failed on smithi016 with status 5: 'sudo systemctl stop ceph-d654e93c-fa46-11ee-bc8e-c7b262605968@mon.a' [7638519] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:20:34 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638519/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638519/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ce4af43caf64c92bdfbef25e95e70e7 Command failed on smithi050 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bced77c0-fa46-11ee-bc8e-c7b262605968 -e sha1=d7e176b41636f150fe82aaedea82def3e9af0203 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' [7638524] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:08:05 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638524/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638524/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d9c791c9b914471b524761d39af0c60 Command failed on smithi047 with status 5: 'sudo systemctl stop ceph- ef7c0120-fa46-11ee-bc8e-c7b262605968@mon.a' [7638527] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:09:05 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638527/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638527/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48492124a00a4b6a9705bf08c1b1ec71 Command failed on smithi106 with status 5: 'sudo systemctl stop ceph-1d447d26-fa47-11ee-bc8e-c7b262605968@mon.a' [7638528] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:11:47 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638528/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638528/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57ca35b1a6d4443dae93a4b075578d40 Command failed on smithi043 with status 5: 'sudo systemctl stop ceph-708f2542-fa49-11ee-bc8e-c7b262605968@mon.a' [7638533] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:08:46 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638533/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638533/ sentry: https://sentry.ceph.com/organizations/ceph/?query=570b49e76bbc45d896a90d889e7c4ee7 Command failed on smithi082 with status 5: 'sudo systemctl stop ceph-d1433484-fa47-11ee-bc8e-c7b262605968@mon.a' [7638534] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:07:59 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638534/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638534/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb6f70400ba84a5aac770089638499f5 Command failed on smithi088 with status 5: 'sudo systemctl stop ceph-0e506220-fa48-11ee-bc8e-c7b262605968@mon.a' [7638537] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:47 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638537/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638537/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b19cc418bc6460898f1bfba8efa443a Command failed (workunit test suites/fsstress.sh) on smithi144 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b3cf8ca51f5d8edaa7cbce710f0de3a66b6f6be9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7638538] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:09:25 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638538/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638538/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88720e919ec34a0ebaf425a5249271c3 Command failed on smithi086 with status 5: 'sudo systemctl stop ceph-4569092e-fa48-11ee-bc8e-c7b262605968@mon.a' [7638545] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:11:13 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638545/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638545/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7adc893031e449fcbb095f420c50278f Command failed on smithi069 with status 5: 'sudo systemctl stop ceph-7042bee6-fa49-11ee-bc8e-c7b262605968@mon.a' [7638547] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:08:39 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638547/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638547/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7456928308f64bad9c8ecc2fde1a9d8f Command failed on smithi027 with status 5: 'sudo systemctl stop ceph-499d0b34-fa49-11ee-bc8e-c7b262605968@mon.a' [7638549] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:38 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638549/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638549/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4740a642b5fd4561b8e1a3050ef70621 Command failed on smithi066 with status 5: 'sudo systemctl stop ceph-2753be7e-fa49-11ee-bc8e-c7b262605968@mon.a' [7638553] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:31 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638553/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638553/ Command failed on smithi029 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.29 deep-scrub' [7638554] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:09:01 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638554/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638554/ sentry: https://sentry.ceph.com/organizations/ceph/?query=345c0abc18db4ee7a3dbb377c4286625 Command failed on smithi049 with status 5: 'sudo systemctl stop ceph-6cea55b0-fa49-11ee-bc8e-c7b262605968@mon.a' [7638556] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:10:55 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638556/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638556/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5db56d59f7304d57bb8d32ca8964e18f Command failed on smithi023 with status 5: 'sudo systemctl stop ceph-6afa8e54-fa4a-11ee-bc8e-c7b262605968@mon.a' [7638558] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:18:48 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638558/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638558/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d2a552b61e146798233fe8053571eba Fuse mount failed to populate/sys/ after 31 seconds [7638559] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:22:18 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638559/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638559/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9dbb6a9e450040989adfd0c4bb0bb647 Command failed on smithi151 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1df6d63a-fa4a-11ee-bc8e-c7b262605968 -e sha1=d7e176b41636f150fe82aaedea82def3e9af0203 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' [7638561] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:08:22 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638561/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638561/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac31b5eaf4654dd38ba294467de49d74 Command failed on smithi090 with status 5: 'sudo systemctl stop ceph- acdb6d0c-f8a1-11ee-bc8c-c7b262605968@mon.a' [7638564] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:08:12 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638564/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638564/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b1e7820c2364ab6bac16c6f73f716a3 Command failed on smithi064 with status 5: 'sudo systemctl stop ceph-7c863100-fa4a-11ee-bc8e-c7b262605968@mon.a' [7638566] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:08:49 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638566/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638566/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c95b8590479411d9db3dd43db688da8 Command failed on smithi080 with status 5: 'sudo systemctl stop ceph-994ee2c8-fa4a-11ee-bc8e-c7b262605968@mon.a' [7638568] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:10:41 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638568/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638568/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f35b8ef3462e4af6aabdc0428e6adcd6 Command failed on smithi033 with status 5: 'sudo systemctl stop ceph-86a26a2c-fa4b-11ee-bc8e-c7b262605968@mon.a' Dead ================================================================= [7638422] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638422/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638422/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7bc5527a357a4e579d7aa610cd1bf305 SSH connection to smithi195 was lost: 'sudo yum upgrade -y linux-firmware' [7638442] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/scrub} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638442/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638442/ Error reimaging machines: Expected smithi096's OS to be centos 9 but found centos 8 [7638443] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638443/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638443/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b856980c2ad4bc9b0ac7870b060ad09 SSH connection to smithi096 was lost: 'sudo yum install -y kernel' [7638474] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638474/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638474/ Error reimaging machines: Expected smithi181's OS to be centos 9 but found centos 8 [7638475] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638475/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638475/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f99889d543b485eab3e2f7e8658167a SSH connection to smithi181 was lost: 'sudo yum install -y kernel' [7638504] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638504/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638504/ Error reimaging machines: Expected smithi029's OS to be ubuntu 22.04 but found centos 9 [7638515] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638515/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638515/ hit max job timeout Pass ================================================================= [7638330] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/client-readahead} time: 00:13:55 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638330/ [7638332] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:40:36 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638332/ [7638333] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:19:01 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638333/ [7638334] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate 3-modules} centos_latest clusters/small-cluster conf/{client mds mgr mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/trim-i24137} time: 00:09:21 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638334/ [7638336] fs/full/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-node-1-mds-1-osd conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/mgr-osd-full} time: 00:15:09 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638336/ [7638337] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client} time: 00:13:39 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638337/ [7638338] fs/mirror/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/mirror} time: 02:00:06 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638338/ [7638339] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate 3-modules} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distro$/{centos_latest} workloads/cephfs-mirror-ha-workunit} time: 00:17:51 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638339/ [7638340] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests} time: 01:00:02 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638340/ [7638341] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:33:50 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638341/ [7638343] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:15:57 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638343/ [7638345] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:45:13 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638345/ [7638346] fs/top/{begin/{0-install 1-ceph 2-logrotate 3-modules} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/fstop} time: 00:11:20 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638346/ [7638348] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/basic}} time: 01:59:20 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638348/ [7638350] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/client-recovery} time: 00:53:17 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638350/ [7638352] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:14:11 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638352/ [7638353] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:23 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638353/ [7638355] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:20:09 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638355/ [7638357] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:17:43 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638357/ [7638358] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} time: 00:11:53 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638358/ [7638359] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/data-scan} time: 00:43:18 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638359/ [7638362] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:10:37 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638362/ [7638363] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/exports} time: 00:51:58 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638363/ [7638364] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:29:42 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638364/ [7638365] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:41:13 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638365/ [7638368] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:19:40 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638368/ [7638373] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:12:26 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638373/ [7638374] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:16:48 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638374/ [7638375] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/fragment} time: 00:21:19 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638375/ [7638376] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:18:26 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638376/ [7638377] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}} time: 00:58:37 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638377/ [7638380] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:21:44 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638380/ [7638381] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:24 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638381/ [7638383] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/ior-shared-file} time: 00:13:32 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638383/ [7638384] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:09:40 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638384/ [7638385] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:49:12 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638385/ [7638388] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/mds-flush} time: 00:09:07 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638388/ [7638389] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:12:20 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638389/ [7638391] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/mds-full} time: 00:25:29 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638391/ [7638393] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:45 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638393/ [7638395] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:50 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638395/ [7638396] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/mds_creation_retry} time: 00:09:41 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638396/ [7638398] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:49:46 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638398/ [7638399] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:12:47 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638399/ [7638400] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/ino_release_cb} time: 00:13:11 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638400/ [7638401] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:16:40 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638401/ [7638403] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/metrics} time: 00:26:43 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638403/ [7638404] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-modules} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{0-client 1-tests/fscrypt-ffsb}} time: 02:27:11 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638404/ [7638405] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/finisher_per_module}} time: 00:13:34 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638405/ [7638407] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:52:58 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638407/ [7638408] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:19:04 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638408/ [7638409] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:11:08 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638409/ [7638410] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/multimds_misc} time: 00:20:29 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638410/ [7638414] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:56:07 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638414/ [7638415] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/openfiletable} time: 00:11:32 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638415/ [7638417] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:15:57 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638417/ [7638418] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:17:54 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638418/ [7638420] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} time: 00:09:59 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638420/ [7638421] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/mdtest} time: 00:21:19 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638421/ [7638423] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:39:31 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638423/ [7638427] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:27:16 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638427/ [7638429] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/iozone}} time: 00:16:07 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638429/ [7638431] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:12:15 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638431/ [7638432] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:12:05 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638432/ [7638433] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quota} time: 00:15:57 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638433/ [7638434] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:10:20 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638434/ [7638437] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/misc}} time: 00:18:36 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638437/ [7638439] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/recovery-fs} time: 00:09:59 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638439/ [7638440] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:13:58 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638440/ [7638445] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:48 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638445/ [7638447] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/sessionmap} time: 00:15:09 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638447/ [7638449] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:41:13 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638449/ [7638450] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:18:56 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638450/ [7638452] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:20:58 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638452/ [7638454] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snap-schedule} time: 02:05:33 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638454/ [7638455] fs/upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/new_ops/{newops}}} time: 00:12:12 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638455/ [7638457] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:16:53 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638457/ [7638458] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap_schedule_snapdir} time: 00:14:43 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638458/ [7638459] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:11:48 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638459/ [7638461] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs/{frag test}} time: 00:28:09 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638461/ [7638462] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cephfs_misc_tests} time: 00:59:42 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638462/ [7638463] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:11:41 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638463/ [7638464] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:11:04 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638464/ [7638466] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 01:08:38 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638466/ [7638468] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:25:09 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638468/ [7638469] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:17:27 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638469/ [7638472] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/snapshots} time: 00:29:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638472/ [7638473] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/snapshot}} time: 00:29:03 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638473/ [7638476] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 00:56:03 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638476/ [7638477] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/strays} time: 00:27:12 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638477/ [7638481] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/test_journal_migration} time: 00:20:40 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638481/ [7638482] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:48:02 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638482/ [7638485] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/truncate_delay} time: 00:13:42 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638485/ [7638486] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:16:17 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638486/ [7638488] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/workunit/dir-max-entries} time: 00:10:33 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638488/ [7638490] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:17:44 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638490/ [7638491] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:13:59 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638491/ [7638492] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:18:27 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638492/ [7638494] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:41:33 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638494/ [7638495] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:48:30 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638495/ [7638499] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:40 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638499/ [7638501] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-3-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/ior-shared-file} time: 00:13:38 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638501/ [7638502] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/multifs-auth} time: 00:22:17 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638502/ [7638503] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 00:34:38 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638503/ [7638506] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/workunit/quota} time: 00:11:04 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638506/ [7638508] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:13:46 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638508/ [7638512] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:19:25 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638512/ [7638513] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:01 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638513/ [7638514] fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:16:24 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638514/ [7638517] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:14:41 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638517/ [7638520] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:14:15 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638520/ [7638521] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:11:34 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638521/ [7638522] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/libcephfs_python} time: 00:10:04 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638522/ [7638523] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_pjd} time: 00:11:22 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638523/ [7638525] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:16:07 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638525/ [7638526] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/asok_dump_tree} time: 00:13:10 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638526/ [7638529] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/auto-repair} time: 00:15:11 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638529/ [7638530] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:19:00 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638530/ [7638531] fs/upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/kernel_untar_build}} time: 00:34:42 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638531/ [7638532] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}} time: 01:04:07 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638532/ [7638535] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/backtrace} time: 00:09:53 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638535/ [7638536] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:43:36 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638536/ [7638539] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:43:02 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638539/ [7638540] fs/traceless/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:27:57 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638540/ [7638541] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-2-client conf/{client mds mgr mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/mdtest} time: 00:16:07 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638541/ [7638542] fs/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health} tasks/failover} time: 01:18:16 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638542/ [7638543] fs/snaps/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/workunit/snaps} time: 01:01:51 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638543/ [7638544] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/cap-flush} time: 00:10:19 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638544/ [7638546] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:13:57 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638546/ [7638548] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/client-limits} time: 00:22:15 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638548/ [7638550] fs/32bits/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_suites_fsstress} time: 00:14:03 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638550/ [7638551] fs/permission/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/fixed-2-ucephfs conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down pg_health} tasks/cfuse_workunit_misc} time: 00:14:34 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638551/ [7638552] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:53:27 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638552/ [7638555] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/client-readahead} time: 00:10:02 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638555/ [7638557] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:18:09 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638557/ [7638560] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/client-recovery} time: 00:49:51 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638560/ [7638562] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:17:10 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638562/ [7638563] fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/finisher_per_module}} time: 00:09:56 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638563/ [7638565] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/damage} time: 00:25:41 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638565/ [7638567] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:53 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638567/ [7638569] fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/data-scan} time: 01:23:06 info: https://pulpito.ceph.com/teuthology-2024-04-03_20:24:15-fs-main-distro-default-smithi/7638569/