2023-06-29T12:43:36.040 INFO:root:teuthology version: 0.0.1.dev134+ge25bf39
2023-06-29T12:43:36.041 INFO:teuthology.results:subset: '23/256'
2023-06-29T12:43:36.041 INFO:teuthology.results:seed: '3567'
2023-06-29T14:37:15.587 INFO:root:teuthology version: 0.0.1.dev134+ge25bf39
2023-06-29T14:37:15.615 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-06-29T19:45:40.777 INFO:teuthology.results:Tests finished! gathering results...
2023-06-29T19:45:43.783 INFO:scrape:Found 49 jobs
2023-06-29T19:45:43.784 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/rishabh-2023-06-29_12:43:28-fs-wip-rishabh-rename-testing-default-smithi/7321335/teuthology.log
2023-06-29T19:45:45.490 WARNING:scrape:Misunderstood line: 
2023-06-29T19:45:50.786 WARNING:scrape:Misunderstood line: 
2023-06-29T19:46:04.087 INFO:scrape:Found 15 distinct failure reasons
2023-06-29T19:46:04.088 INFO:scrape:Dead: None
2023-06-29T19:46:04.088 INFO:scrape:13 jobs: ['7321335', '7321358', '7321321', '7321323', '7321331', '7321340', '7321353', '7321344', '7321351', '7321360', '7321359', '7321350', '7321356']
2023-06-29T19:46:04.088 INFO:scrape:
2023-06-29T19:46:04.089 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c403ab5c815eaf908a3d8b31cb0ce62cdd9db36 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-06-29T19:46:04.089 INFO:scrape:1 jobs: ['7321343']
2023-06-29T19:46:04.090 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}']
2023-06-29T19:46:04.090 INFO:scrape:
2023-06-29T19:46:04.090 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/pjd.sh
2023-06-29T19:46:04.090 INFO:scrape:8 jobs: ['7321329', '7321342', '7321313', '7321319', '7321320', '7321349', '7321341', '7321328']
2023-06-29T19:46:04.091 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}']
2023-06-29T19:46:04.092 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'frag', 'fs/32bits/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/{0-subvolume/{with-quota}']
2023-06-29T19:46:04.092 INFO:scrape:
2023-06-29T19:46:04.092 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable)
2023-06-29T19:46:04.092 INFO:scrape:2 jobs: ['7321317', '7321334']
2023-06-29T19:46:04.093 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2023-06-29T19:46:04.093 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2023-06-29T19:46:04.093 INFO:scrape:
2023-06-29T19:46:04.094 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/ffsb.sh
2023-06-29T19:46:04.094 INFO:scrape:1 jobs: ['7321337']
2023-06-29T19:46:04.094 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_ffsb', 'traceless/50pc}']
2023-06-29T19:46:04.095 INFO:scrape:
2023-06-29T19:46:04.095 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds
2023-06-29T19:46:04.095 INFO:scrape:3 jobs: ['7321332', '7321336', '7321316']
2023-06-29T19:46:04.096 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay']
2023-06-29T19:46:04.096 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}']
2023-06-29T19:46:04.096 INFO:scrape:
2023-06-29T19:46:04.097 INFO:scrape:Failure: Command failed on smithi052 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4c403ab5c815eaf908a3d8b31cb0ce62cdd9db36 pull'
2023-06-29T19:46:04.097 INFO:scrape:4 jobs: ['7321326', '7321346', '7321354', '7321348']
2023-06-29T19:46:04.097 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}']
2023-06-29T19:46:04.098 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distros/ubuntu_latest', 'frag', 'fs/multiclient/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2023-06-29T19:46:04.098 INFO:scrape:
2023-06-29T19:46:04.098 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/dbench.sh
2023-06-29T19:46:04.099 INFO:scrape:1 jobs: ['7321325']
2023-06-29T19:46:04.099 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_dbench', 'traceless/50pc}']
2023-06-29T19:46:04.099 INFO:scrape:
2023-06-29T19:46:04.100 INFO:scrape:Failure: Command failed on smithi144 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-06-29T19:46:04.100 INFO:scrape:1 jobs: ['7321327']
2023-06-29T19:46:04.100 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/mdtest}']
2023-06-29T19:46:04.100 INFO:scrape:
2023-06-29T19:46:04.101 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2023-06-29T19:46:04.101 INFO:scrape:1 jobs: ['7321338']
2023-06-29T19:46:04.102 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']
2023-06-29T19:46:04.102 INFO:scrape:
2023-06-29T19:46:04.103 INFO:scrape:Failure: Command failed on smithi084 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f17ceb2-167f-11ee-9b2e-001a4aab830c -e sha1=4c403ab5c815eaf908a3d8b31cb0ce62cdd9db36 -- bash -c 'ceph orch ps'"
2023-06-29T19:46:04.113 INFO:scrape:1 jobs: ['7321330']
2023-06-29T19:46:04.113 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific']
2023-06-29T19:46:04.113 INFO:scrape:
2023-06-29T19:46:04.114 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-06-29T19:46:04.114 INFO:scrape:1 jobs: ['7321347']
2023-06-29T19:46:04.114 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_fsstress', 'traceless/50pc}']
2023-06-29T19:46:04.115 INFO:scrape:
2023-06-29T19:46:04.115 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c403ab5c815eaf908a3d8b31cb0ce62cdd9db36 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2023-06-29T19:46:04.115 INFO:scrape:1 jobs: ['7321339']
2023-06-29T19:46:04.116 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}']
2023-06-29T19:46:04.116 INFO:scrape:
2023-06-29T19:46:04.116 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4c403ab5c815eaf908a3d8b31cb0ce62cdd9db36 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh'
2023-06-29T19:46:04.117 INFO:scrape:1 jobs: ['7321352']
2023-06-29T19:46:04.117 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}']
2023-06-29T19:46:04.117 INFO:scrape:
2023-06-29T19:46:04.117 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/client/test.sh
2023-06-29T19:46:04.118 INFO:scrape:1 jobs: ['7321315']
2023-06-29T19:46:04.118 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client}']
2023-06-29T19:46:04.119 INFO:scrape: