2023-11-04T04:31:44.112 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T04:31:44.113 INFO:teuthology.results:subset: '332/512'
2023-11-04T04:31:44.113 INFO:teuthology.results:seed: '9821'
2023-11-04T07:34:05.538 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T07:34:05.569 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-04T17:45:00.220 INFO:teuthology.results:Tests finished! gathering results...
2023-11-04T17:45:15.577 INFO:scrape:Found 222 jobs
2023-11-04T17:45:15.579 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447046/teuthology.log
2023-11-04T17:45:19.658 WARNING:scrape:Misunderstood line: 
2023-11-04T17:45:22.460 WARNING:scrape:Misunderstood line: 
2023-11-04T17:45:22.461 INFO:scrape:Found 9 distinct failure reasons
2023-11-04T17:45:22.462 INFO:scrape:Dead: None
2023-11-04T17:45:22.462 INFO:scrape:50 jobs
2023-11-04T17:45:22.463 INFO:scrape:
2023-11-04T17:45:22.463 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-11-04T17:45:22.463 INFO:scrape:3 jobs: ['7447099', '7447200', '7447069']
2023-11-04T17:45:22.464 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay']
2023-11-04T17:45:22.464 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/dbench}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2023-11-04T17:45:22.465 INFO:scrape:
2023-11-04T17:45:22.465 INFO:scrape:Dead: b'2023-11-04T17:16:38.690 DEBUG:teuthology.exit:Finished running handlers'
2023-11-04T17:45:22.466 INFO:scrape:ceph version 18.0.0-7050-g826ae420 (826ae4205210243850fc3f7bc3f8aea0d1f5ced2) reef (dev)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd7) [0x557c4c41edc1]
 2: (KernelDevice::_aio_thread()+0x18a6) [0x557c4d095f96]
 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x557c4d09df91]
 4: /lib64/libpthread.so.0(+0x81cf) [0x7f1f764b81cf]
2023-11-04T17:45:22.466 INFO:scrape:1 jobs: ['7447118']
2023-11-04T17:45:22.466 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}']
2023-11-04T17:45:22.467 INFO:scrape:
2023-11-04T17:45:22.467 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi184 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=826ae4205210243850fc3f7bc3f8aea0d1f5ced2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2023-11-04T17:45:22.468 INFO:scrape:2 jobs: ['7447173', '7447249']
2023-11-04T17:45:22.468 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'ubuntu_latest}}']
2023-11-04T17:45:22.468 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'ubuntu_latest}}']
2023-11-04T17:45:22.469 INFO:scrape:
2023-11-04T17:45:22.470 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh
2023-11-04T17:45:22.470 INFO:scrape:1 jobs: ['7447198']
2023-11-04T17:45:22.471 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy']
2023-11-04T17:45:22.471 INFO:scrape:
2023-11-04T17:45:22.471 INFO:scrape:Failure: Command failed on smithi076 with status 5: 'sudo systemctl stop ceph-26cb3e7a-7ad1-11ee-8db9-212e2dc638e7@mon.b'
2023-11-04T17:45:22.472 INFO:scrape:1 jobs: ['7447104']
2023-11-04T17:45:22.472 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}']
2023-11-04T17:45:22.472 INFO:scrape:
2023-11-04T17:45:22.473 INFO:scrape:Failure: Command failed on smithi137 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:826ae4205210243850fc3f7bc3f8aea0d1f5ced2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a09c629c-7ad6-11ee-8db9-212e2dc638e7 -- bash -c 'ceph fs volume rename foo bar --yes-i-really-mean-it'"
2023-11-04T17:45:22.473 INFO:scrape:3 jobs: ['7447149', '7447263', '7447218']
2023-11-04T17:45:22.474 INFO:scrape:suites intersection: []
2023-11-04T17:45:22.474 INFO:scrape:suites union: ['1-ranks/1', '1-rename', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'distro/single-container-host', 'fail_fs/yes', 'fs/cephadm/renamevolume/{0-start', 'fs/nfs/{cluster/{1-node}', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/ignorelist_health', 'overrides/ignorelist_health}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'supported-random-distros$/{centos_latest}', 'syntax}', 'tasks/nfs}', 'tasks/{0-from/quincy']
2023-11-04T17:45:22.475 INFO:scrape:
2023-11-04T17:45:22.475 INFO:scrape:Valgrind: client (Leak_StillReachable), mon (Leak_StillReachable)
2023-11-04T17:45:22.475 INFO:scrape:2 jobs: ['7447138', '7447057']
2023-11-04T17:45:22.476 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}']
2023-11-04T17:45:22.476 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}']
2023-11-04T17:45:22.476 INFO:scrape:
2023-11-04T17:45:22.477 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2023-11-04T17:45:22.477 INFO:scrape:1 jobs: ['7447237']
2023-11-04T17:45:22.478 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']
2023-11-04T17:45:22.478 INFO:scrape: