2023-04-11T07:53:33.057 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-11T07:53:33.059 INFO:teuthology.results:subset: '203/256'
2023-04-11T07:53:33.059 INFO:teuthology.results:seed: '8747'
2023-04-11T11:26:44.728 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-11T11:26:44.761 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-11T20:38:25.948 INFO:teuthology.results:Tests finished! gathering results...
2023-04-11T20:38:33.456 INFO:scrape:Found 82 jobs
2023-04-11T20:38:47.748 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/rishabh-2023-04-11_07:53:16-fs-wip-rishabh-2023Apr8-2003-testing-default-smithi/7238009/teuthology.log
2023-04-11T20:39:24.905 WARNING:scrape:Misunderstood line: 
2023-04-11T20:39:27.486 INFO:scrape:Found 26 distinct failure reasons
2023-04-11T20:39:27.487 INFO:scrape:Failure: Cannot connect to remote host smithi038
2023-04-11T20:39:27.488 INFO:scrape:1 jobs: ['7238039']
2023-04-11T20:39:27.488 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_blogbench', 'traceless/50pc}']
2023-04-11T20:39:27.488 INFO:scrape:
2023-04-11T20:39:27.489 INFO:scrape:Failure: Command failed on smithi002 with status 1: 'sudo yum install -y kernel'
2023-04-11T20:39:27.489 INFO:scrape:16 jobs: ['7238131', '7238123', '7238121', '7238120', '7238113', '7238122', '7238064', '7238130', '7238117', '7238132', '7238126', '7238128', '7238041', '7238129', '7238124', '7238133']
2023-04-11T20:39:27.490 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2023-04-11T20:39:27.490 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-client', '1-mount/mount/fuse', '1-tests/blogbench}}}', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-upgrade', '2-workload/stress_tests/{0-client-upgrade', '2-workunit/cfuse_workunit_suites_fsstress}}', '3-compat_client/quincy}}', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'centos_latest', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client', 'clusters/1-mds-2-client-micro', 'clusters/1-mds-3-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/fixed-2-ucephfs', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'frag', 'fs/32bits/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/osd-mds-delay', 'multifs', 'multimds/yes', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/ior-shared-file}', 'tasks/libcephfs_python}', 'tasks/mdtest}', 'tasks/{0-from/nautilus', 'tasks/{0-octopus', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'tasks/{1-thrash/mds', 'thrashosds-health}', 'wsync/no}', 'wsync/yes}']
2023-04-11T20:39:27.490 INFO:scrape:
2023-04-11T20:39:27.491 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-04-11T20:39:27.491 INFO:scrape:4 jobs: ['7238108', '7238054', '7238115', '7238111']
2023-04-11T20:39:27.491 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'rhel_8}', 'session_timeout}', 'standby-replay', 'wsync/yes}']
2023-04-11T20:39:27.492 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/postgres}}', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}']
2023-04-11T20:39:27.492 INFO:scrape:
2023-04-11T20:39:27.492 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/client/test.sh
2023-04-11T20:39:27.493 INFO:scrape:1 jobs: ['7238013']
2023-04-11T20:39:27.493 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client}']
2023-04-11T20:39:27.493 INFO:scrape:
2023-04-11T20:39:27.493 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/pjd.sh
2023-04-11T20:39:27.494 INFO:scrape:7 jobs: ['7238094', '7238076', '7238074', '7238033', '7238096', '7238031', '7238072']
2023-04-11T20:39:27.494 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}']
2023-04-11T20:39:27.494 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'frag', 'fs/32bits/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/{0-subvolume/{with-quota}']
2023-04-11T20:39:27.495 INFO:scrape:
2023-04-11T20:39:27.495 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b923eefb99088d6ebb1fa03c93819d48781dab9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2023-04-11T20:39:27.495 INFO:scrape:2 jobs: ['7238088', '7238068']
2023-04-11T20:39:27.496 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2023-04-11T20:39:27.496 INFO:scrape:suites union: ['1-ceph', '1-mount/mount/fuse', '1-tests/dbench}}}', '2-logrotate}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/fixed-2-ucephfs', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro/{centos_8}', 'fs/traceless/{begin/{0-install', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/cfuse_workunit_suites_dbench', 'tasks/{0-from/nautilus', 'traceless/50pc}']
2023-04-11T20:39:27.496 INFO:scrape:
2023-04-11T20:39:27.496 INFO:scrape:Failure: reached maximum tries (30) after waiting for 300 seconds
2023-04-11T20:39:27.497 INFO:scrape:5 jobs: ['7238112', '7238116', '7238110', '7238109', '7238107']
2023-04-11T20:39:27.497 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2023-04-11T20:39:27.497 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-client', '1-mount/mount/fuse', '1-tests/iozone}}}', '2-logrotate}', '2-scrub/no', '2-upgrade', '2-workload/stress_tests/{0-client-upgrade', '2-workunit/iozone}}', '3-client-upgrade', '3-snaps/yes', '4-compat_client', '4-flush/no', '5-client-sanity}}', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/fixed-2-ucephfs', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro/{centos_8}', 'frag', 'fs/permission/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'msgr-failures/osd-mds-delay', 'multifs', 'multimds/no', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_misc}', 'tasks/{0-from/pacific', 'tasks/{0-octopus', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'wsync/no}']
2023-04-11T20:39:27.498 INFO:scrape:
2023-04-11T20:39:27.498 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-04-11T20:39:27.498 INFO:scrape:3 jobs: ['7238099', '7238119', '7238056']
2023-04-11T20:39:27.499 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}']
2023-04-11T20:39:27.499 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_20.04}', 'frag', 'fs/traceless/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{ignorelist_health', 'ranks/5', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/fsstress', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'traceless/50pc}', 'validater/valgrind}', 'wsync/yes}']
2023-04-11T20:39:27.499 INFO:scrape:
2023-04-11T20:39:27.499 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/ffsb.sh
2023-04-11T20:39:27.500 INFO:scrape:2 jobs: ['7238027', '7238101']
2023-04-11T20:39:27.500 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'mds', 'mon', 'osd}']
2023-04-11T20:39:27.500 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{rhel_8}', 'frag', 'fs/traceless/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/{0-subvolume/{with-no-extra-options}', 'traceless/50pc}', 'wsync/yes}']
2023-04-11T20:39:27.501 INFO:scrape:
2023-04-11T20:39:27.501 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b923eefb99088d6ebb1fa03c93819d48781dab9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh'
2023-04-11T20:39:27.501 INFO:scrape:1 jobs: ['7238090']
2023-04-11T20:39:27.501 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}']
2023-04-11T20:39:27.502 INFO:scrape:
2023-04-11T20:39:27.502 INFO:scrape:Failure: Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)
2023-04-11T20:39:27.502 INFO:scrape:3 jobs: ['7238114', '7238017', '7238066']
2023-04-11T20:39:27.503 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2023-04-11T20:39:27.503 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/multifs-auth}']
2023-04-11T20:39:27.503 INFO:scrape:
2023-04-11T20:39:27.504 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi175 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b923eefb99088d6ebb1fa03c93819d48781dab9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2023-04-11T20:39:27.504 INFO:scrape:1 jobs: ['7238045']
2023-04-11T20:39:27.504 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/fsstress}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/nautilus']
2023-04-11T20:39:27.504 INFO:scrape:
2023-04-11T20:39:27.505 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b923eefb99088d6ebb1fa03c93819d48781dab9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2023-04-11T20:39:27.505 INFO:scrape:3 jobs: ['7238046', '7238106', '7238011']
2023-04-11T20:39:27.505 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'n/5', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay']
2023-04-11T20:39:27.506 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}', 'wsync/yes}']
2023-04-11T20:39:27.506 INFO:scrape:
2023-04-11T20:39:27.506 INFO:scrape:Dead: None
2023-04-11T20:39:27.507 INFO:scrape:1 jobs: ['7238009']
2023-04-11T20:39:27.507 INFO:scrape:
2023-04-11T20:39:27.507 INFO:scrape:Failure: "1681202476.4629526 mon.a (mon.0) 1655 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY)" in cluster log
2023-04-11T20:39:27.507 INFO:scrape:1 jobs: ['7238029']
2023-04-11T20:39:27.508 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/metrics}']
2023-04-11T20:39:27.508 INFO:scrape:
2023-04-11T20:39:27.508 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b923eefb99088d6ebb1fa03c93819d48781dab9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-04-11T20:39:27.509 INFO:scrape:1 jobs: ['7238097']
2023-04-11T20:39:27.509 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}']
2023-04-11T20:39:27.509 INFO:scrape:
2023-04-11T20:39:27.509 INFO:scrape:Dead: b"TypeError: argument of type 'NoneType' is not iterable"
2023-04-11T20:39:27.510 INFO:scrape:2 jobs: ['7238084', '7238134']
2023-04-11T20:39:27.510 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}']
2023-04-11T20:39:27.510 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/xfstests-dev}', 'traceless/50pc}']
2023-04-11T20:39:27.511 INFO:scrape:
2023-04-11T20:39:27.511 INFO:scrape:Failure: more than one file system available
2023-04-11T20:39:27.511 INFO:scrape:6 jobs: ['7238080', '7238037', '7238103', '7238060', '7238125', '7238023']
2023-04-11T20:39:27.511 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health}']
2023-04-11T20:39:27.512 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-04-11T20:39:27.512 INFO:scrape:
2023-04-11T20:39:27.512 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi107 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b923eefb99088d6ebb1fa03c93819d48781dab9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2023-04-11T20:39:27.513 INFO:scrape:2 jobs: ['7238007', '7238062']
2023-04-11T20:39:27.513 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-04-11T20:39:27.513 INFO:scrape:suites union: ['1-ceph', '1-mount/mount/fuse', '1-tests/kernel_untar_build}}}', '2-logrotate}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/1a3s-mds-2c-client', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'pg-warn}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{0-from/pacific']
2023-04-11T20:39:27.514 INFO:scrape:
2023-04-11T20:39:27.514 INFO:scrape:Dead: b'2023-04-11T20:36:00.486 DEBUG:teuthology.exit:Finished running handlers'
2023-04-11T20:39:27.514 INFO:scrape:1 jobs: ['7238058']
2023-04-11T20:39:27.514 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}']
2023-04-11T20:39:27.515 INFO:scrape:
2023-04-11T20:39:27.515 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2023-04-11T20:39:27.515 INFO:scrape:1 jobs: ['7238043']
2023-04-11T20:39:27.516 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']
2023-04-11T20:39:27.516 INFO:scrape:
2023-04-11T20:39:27.516 INFO:scrape:Failure: SSH connection to smithi017 was lost: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900'"
2023-04-11T20:39:27.516 INFO:scrape:1 jobs: ['7238021']
2023-04-11T20:39:27.517 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2023-04-11T20:39:27.517 INFO:scrape:
2023-04-11T20:39:27.517 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-04-11T20:39:27.518 INFO:scrape:1 jobs: ['7238048']
2023-04-11T20:39:27.518 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/quota}']
2023-04-11T20:39:27.518 INFO:scrape:
2023-04-11T20:39:27.518 INFO:scrape:Failure: Command failed on smithi177 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4d761044-d852-11ed-9aff-001a4aab830c -- ceph mon dump -f json'
2023-04-11T20:39:27.519 INFO:scrape:1 jobs: ['7238050']
2023-04-11T20:39:27.519 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4']
2023-04-11T20:39:27.519 INFO:scrape:
2023-04-11T20:39:27.520 INFO:scrape:Timeout 120 running ceph --cluster ceph tell mds.1:0 scrub status
2023-04-11T20:39:27.520 INFO:scrape:1 jobs: ['7238092']
2023-04-11T20:39:27.520 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}']
2023-04-11T20:39:27.520 INFO:scrape:
2023-04-11T20:39:27.521 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable)
2023-04-11T20:39:27.521 INFO:scrape:1 jobs: ['7238127']
2023-04-11T20:39:27.521 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2023-04-11T20:39:27.521 INFO:scrape: