2024-02-14T12:35:40.877 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-14T12:35:40.877 INFO:teuthology.results:subset: '960/1024' 2024-02-14T12:35:40.877 INFO:teuthology.results:seed: '962' 2024-02-14T16:34:37.254 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-14T16:34:37.272 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-14T17:18:40.748 INFO:teuthology.results:Tests finished! gathering results... 2024-02-14T17:18:46.953 INFO:scrape:Found 67 jobs 2024-02-14T17:18:50.656 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:50.656 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:50.656 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:50.656 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:51.200 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:51.201 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:51.201 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:51.201 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:51.709 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:51.709 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:51.709 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:51.709 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:52.226 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:52.226 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:52.226 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:52.226 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:52.743 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:52.744 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:52.744 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:52.744 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:53.257 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:53.258 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:53.258 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:53.258 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:53.780 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:53.781 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:53.781 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:53.781 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.156 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.157 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.157 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.157 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.161 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.161 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.161 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.161 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.165 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.166 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.166 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.166 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.170 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.170 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.170 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.170 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.208 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.208 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.208 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.208 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.333 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.333 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.333 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.333 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.337 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.338 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.338 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.338 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.342 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.342 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.342 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.342 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.347 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.347 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.347 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.347 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.351 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.352 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.352 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.352 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.466 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.467 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.467 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.467 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.471 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.471 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.471 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.472 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.476 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.476 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.476 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.476 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.480 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.481 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.481 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.482 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.486 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.486 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.486 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.486 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.601 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.602 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.602 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.602 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.606 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.606 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.606 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.607 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.611 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.611 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.611 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.611 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.615 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.615 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.615 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.616 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.620 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.620 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.620 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.620 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.733 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.733 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.733 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.733 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.737 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.738 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.738 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.738 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.742 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.742 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.742 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.742 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.746 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.746 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.746 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.747 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.751 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.751 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.751 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.751 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.865 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.865 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.865 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.866 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.870 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.870 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.870 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.870 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.874 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.874 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.874 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.874 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.878 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.879 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.879 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.879 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.883 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.883 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.883 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:55.883 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.005 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.005 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.005 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.006 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.010 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.010 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.010 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.010 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.015 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.015 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.015 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.015 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.020 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.020 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.020 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.020 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.025 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.025 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.025 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.025 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.140 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.140 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.141 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.141 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.146 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.147 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.147 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.147 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.151 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.151 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.151 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.151 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.156 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.156 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.156 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.156 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.160 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.160 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.160 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.160 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.273 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.273 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.273 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.274 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.278 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.278 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.278 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.278 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.282 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.282 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.282 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.282 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.287 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.287 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.287 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.287 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.291 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.291 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.291 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:18:56.291 WARNING:scrape:Saw end of BT but not start 2024-02-14T17:19:02.745 INFO:scrape:Found 10 distinct failure reasons 2024-02-14T17:19:02.745 INFO:scrape:Failure: local variable 'version' referenced before assignment 2024-02-14T17:19:02.745 INFO:scrape:48 jobs 2024-02-14T17:19:02.746 INFO:scrape:suites intersection: [] 2024-02-14T17:19:02.747 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-client', '1-mount', '1-ranks/1', '1-ranks/2', '1-rename', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-upgrade', '2-workload/dbench', '3-compat_client/quincy}}', '3-inline/no', '3-inline/yes', '3-snaps/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream', 'centos_9.stream', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'debug', 'distro/centos_latest', 'distro/single-container-host', 'distro/single-container-host}', 'distro/{centos_latest}', 'export-check', 'fail_fs/no', 'fail_fs/yes', 'frag', 'fs/cephadm/multivolume/{0-start', 'fs/cephadm/renamevolume/{0-start', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/nfs/{cluster/{1-node}', 'fs/shell/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'multimds/no', 'multimds/yes', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/ignorelist_health', 'overrides/ignorelist_health}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn', 'pg-warn}', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'roles', 'session_timeout}', 'standby-replay', 'subvol_versions/create_subvol_version_v1', 'supported-random-distros$/{centos_latest}', 'syntax}', 'tasks/cephfs-shell}', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/dbench', 'tasks/mirror}}', 'tasks/multifs-auth}', 'tasks/nfs}', 'tasks/sessionmap}', 'tasks/{0-from/quincy', 'tasks/{0-octopus', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'traceless/50pc}', 'validater/valgrind}', 'wsync/no}'] 2024-02-14T17:19:02.747 INFO:scrape: 2024-02-14T17:19:02.747 INFO:scrape:Failure: "2024-02-14T12:57:28.149442+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log 2024-02-14T17:19:02.747 INFO:scrape:3 jobs: ['7559583', '7559564', '7559595'] 2024-02-14T17:19:02.747 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2024-02-14T17:19:02.747 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/norstats}}', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2024-02-14T17:19:02.747 INFO:scrape: 2024-02-14T17:19:02.748 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi016 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6fe3fa3adb3a69ed9af8a494c4670cfd0d57fe0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-02-14T17:19:02.748 INFO:scrape:1 jobs: ['7559628'] 2024-02-14T17:19:02.748 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-02-14T17:19:02.748 INFO:scrape: 2024-02-14T17:19:02.748 INFO:scrape:Failure: valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions 2024-02-14T17:19:02.748 INFO:scrape:2 jobs: ['7559596', '7559571'] 2024-02-14T17:19:02.748 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-02-14T17:19:02.748 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-02-14T17:19:02.748 INFO:scrape: 2024-02-14T17:19:02.749 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-02-14T17:19:02.749 INFO:scrape:1 jobs: ['7559614'] 2024-02-14T17:19:02.749 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2024-02-14T17:19:02.749 INFO:scrape: 2024-02-14T17:19:02.749 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev), test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-02-14T17:19:02.749 INFO:scrape:1 jobs: ['7559578'] 2024-02-14T17:19:02.749 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/xfstests-dev}'] 2024-02-14T17:19:02.749 INFO:scrape: 2024-02-14T17:19:02.749 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6fe3fa3adb3a69ed9af8a494c4670cfd0d57fe0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-02-14T17:19:02.749 INFO:scrape:1 jobs: ['7559621'] 2024-02-14T17:19:02.750 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2024-02-14T17:19:02.750 INFO:scrape: 2024-02-14T17:19:02.750 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-02-14T17:19:02.750 INFO:scrape:1 jobs: ['7559617'] 2024-02-14T17:19:02.750 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/pool-perm}'] 2024-02-14T17:19:02.750 INFO:scrape: 2024-02-14T17:19:02.750 INFO:scrape:Failure: SSH connection to smithi196 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2024-02-14T17:19:02.750 INFO:scrape:1 jobs: ['7559576'] 2024-02-14T17:19:02.750 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-dbench}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2024-02-14T17:19:02.750 INFO:scrape: 2024-02-14T17:19:02.751 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-02-14T17:19:02.751 INFO:scrape:1 jobs: ['7559581'] 2024-02-14T17:19:02.751 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-02-14T17:19:02.751 INFO:scrape: 2024-02-14T17:19:02.755 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/ failed: 60 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7559562] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559562/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559562/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e6531700ad14088909f63c464fa560a local variable 'version' referenced before assignment [7559563] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/sessionmap} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559563/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559563/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55aac39efed3476d94cecfd0cea60750 local variable 'version' referenced before assignment [7559564] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:23:32 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559564/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559564/ "2024-02-14T12:54:37.806388+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7559565] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559565/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559565/ sentry: https://sentry.ceph.com/organizations/ceph/?query=644c02e8a9de429daed328b57fdb2b7c local variable 'version' referenced before assignment [7559568] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559568/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559568/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4f049449c694da9b83f7c4662141ad7 local variable 'version' referenced before assignment [7559569] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559569/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559569/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d057dbbd843e45059bfe63eab41dc989 local variable 'version' referenced before assignment [7559570] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559570/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559570/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f02874f6d18946fbb8a90d2a6f6c49c9 local variable 'version' referenced before assignment [7559571] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:24:22 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559571/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559571/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7559572] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559572/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559572/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8371a2e39a9741d28a72b91d03e8ee83 local variable 'version' referenced before assignment [7559574] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559574/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559574/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b867c5671b7749079e118991b755e8e0 local variable 'version' referenced before assignment [7559575] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559575/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559575/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89d9bc6649f149929c58ba5bcb2f819d local variable 'version' referenced before assignment [7559576] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} ----------------------------------------------------------------- time: 03:53:38 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559576/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559576/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a7867534c3bc4926b8d2b8d86bb24bd8 SSH connection to smithi196 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [7559577] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559577/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559577/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d889a0a31e2e43709dc02e02c6cf4b1d local variable 'version' referenced before assignment [7559578] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/xfstests-dev} ----------------------------------------------------------------- time: 03:57:18 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559578/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559578/ sentry: https://sentry.ceph.com/organizations/ceph/?query=022716e6df214b918ed3d83038decfbc Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev), test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7559579] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559579/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559579/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88fc17f77c644c9d9d8df425c0a8cc0b local variable 'version' referenced before assignment [7559580] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559580/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559580/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce3c4b385e934650be2a82b48f3253be local variable 'version' referenced before assignment [7559581] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:13:25 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559581/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559581/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1a893005fa54da2a94d83db57fe7696 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7559582] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559582/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559582/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac9bc771484649188abccbeefa3f10a8 local variable 'version' referenced before assignment [7559583] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:22:49 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559583/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559583/ "2024-02-14T12:57:28.149442+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7559584] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559584/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559584/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1076aea2ac1c4f28a123fd8b91df2b69 local variable 'version' referenced before assignment [7559585] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559585/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559585/ sentry: https://sentry.ceph.com/organizations/ceph/?query=10291afc40ab498da4691996cb08f3bd local variable 'version' referenced before assignment [7559586] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559586/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559586/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2c68005d15f4007abee5546fae35eca local variable 'version' referenced before assignment [7559587] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559587/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559587/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e744bb83d234909a894aa231ae569ff local variable 'version' referenced before assignment [7559588] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559588/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559588/ sentry: https://sentry.ceph.com/organizations/ceph/?query=169513174f6a4f90bc2251685b1b4ea1 local variable 'version' referenced before assignment [7559590] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559590/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559590/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07e731cc5722469fba0b9a842e8be569 local variable 'version' referenced before assignment [7559591] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559591/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559591/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61f0458cfd9b4cac8c73b6e3e8be2e2b local variable 'version' referenced before assignment [7559592] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559592/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559592/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b18e19ef96643d89d2db42a89f66418 local variable 'version' referenced before assignment [7559593] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559593/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559593/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61c3fb1de1bf40d0897df97d14d93558 local variable 'version' referenced before assignment [7559594] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559594/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559594/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed4aa5288ac443cdb1e694c283cc7b02 local variable 'version' referenced before assignment [7559595] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:23:56 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559595/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559595/ "2024-02-14T13:00:46.775274+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log [7559596] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:22:58 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559596/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559596/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7559597] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559597/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559597/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d03cf363b8324438ad8679629563855d local variable 'version' referenced before assignment [7559598] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559598/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559598/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b97a0477057f4426ad0aa118bd86c8ce local variable 'version' referenced before assignment [7559599] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559599/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559599/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd19fc967e314a878833c84562994ea6 local variable 'version' referenced before assignment [7559600] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559600/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559600/ sentry: https://sentry.ceph.com/organizations/ceph/?query=014946980ac54cd6ac9663f263529057 local variable 'version' referenced before assignment [7559601] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559601/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559601/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f54881fb015143859a79086039c9648a local variable 'version' referenced before assignment [7559602] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559602/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559602/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0fb86fc6ea67487abd261202d7a8cc1f local variable 'version' referenced before assignment [7559603] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559603/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559603/ sentry: https://sentry.ceph.com/organizations/ceph/?query=656e76f1fd3b403e977240834e50b6c8 local variable 'version' referenced before assignment [7559604] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559604/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559604/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eee4de09472b4f6e8e2347ab898aa12b local variable 'version' referenced before assignment [7559607] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559607/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559607/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aee47a9dfd634c6f8f448c015c0d511b local variable 'version' referenced before assignment [7559608] fs/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559608/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559608/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2566f47e4d2942a89e49e0007e5dc957 local variable 'version' referenced before assignment [7559609] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559609/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559609/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78254c85be3c461ebf1ab375fa7cb344 local variable 'version' referenced before assignment [7559610] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559610/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559610/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f47df1ebe014409e8884eae6a9345aa2 local variable 'version' referenced before assignment [7559611] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559611/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559611/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd7de823390841789914cf87a107c6df local variable 'version' referenced before assignment [7559612] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559612/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559612/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abaa0f83dfe14244ad423a7609f6b3a2 local variable 'version' referenced before assignment [7559613] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559613/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559613/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcc86fbbc0324673859d3b85f096a58a local variable 'version' referenced before assignment [7559614] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:11:25 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559614/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559614/ error during scrub thrashing: rank damage found: {'backtrace'} [7559615] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559615/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559615/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4739e1ad18194122a66bcddca96e7c88 local variable 'version' referenced before assignment [7559616] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559616/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559616/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d300e778b3b4d1e82d8625ed6cc7306 local variable 'version' referenced before assignment [7559617] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} ----------------------------------------------------------------- time: 00:15:05 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559617/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559617/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5340856635c34687a486047624b72077 Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7559618] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559618/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559618/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2ee96896fc44e378d1e30c2c6be85f7 local variable 'version' referenced before assignment [7559620] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559620/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559620/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bf402da093bd419081fd712a479d0914 local variable 'version' referenced before assignment [7559621] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:50:04 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559621/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559621/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e555d8617d2541938f36130109b2d731 Command failed (workunit test suites/dbench.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6fe3fa3adb3a69ed9af8a494c4670cfd0d57fe0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7559622] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559622/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559622/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98f5ec7693ae45548cae247eda1c13fe local variable 'version' referenced before assignment [7559623] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559623/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559623/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9512dc86e46d495691fc58c2ba417869 local variable 'version' referenced before assignment [7559624] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559624/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559624/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a65c433ef08047bcaf0ea0d4232b7509 local variable 'version' referenced before assignment [7559625] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559625/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559625/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ead5f6f4bbb848d78f1103b3bca90022 local variable 'version' referenced before assignment [7559626] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559626/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559626/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd5bf6f693154a2bb0a5fbd87f0413ab local variable 'version' referenced before assignment [7559627] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559627/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559627/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d4ce4f467da4dbe8239eefd503e7ae6 local variable 'version' referenced before assignment [7559628] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:34:35 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559628/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559628/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc103d6fd0be4b61b78260d86e5e6ac7 Command failed (workunit test suites/fsx.sh) on smithi016 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a6fe3fa3adb3a69ed9af8a494c4670cfd0d57fe0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' Pass ================================================================= [7559566] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:40:46 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559566/ [7559567] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:36:01 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559567/ [7559573] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/test_journal_migration} time: 00:15:37 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559573/ [7559589] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:35:08 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559589/ [7559605] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds-full} time: 00:18:56 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559605/ [7559606] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:20 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559606/ [7559619] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:12:56 info: https://pulpito.ceph.com/vshankar-2024-02-14_05:24:17-fs-wip-vshankar-testing-20240206.051942-testing-default-smithi/7559619/