2022-06-21T12:14:42.502 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0
2022-06-21T12:14:42.504 INFO:teuthology.results:subset: '25/32'
2022-06-21T12:14:42.505 INFO:teuthology.results:seed: '657'
2022-06-21T12:15:56.938 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0
2022-06-21T12:15:56.945 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-06-21T16:09:16.318 INFO:teuthology.results:Tests finished! gathering results...
2022-06-21T16:09:17.415 INFO:scrape:Found 12 jobs
2022-06-21T16:09:21.854 INFO:scrape:Found 6 distinct failure reasons
2022-06-21T16:09:21.855 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds
2022-06-21T16:09:21.856 INFO:scrape:1 jobs: ['6889289']
2022-06-21T16:09:21.856 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-check-counter', 'workunit/kernel_untar_build}}']
2022-06-21T16:09:21.857 INFO:scrape:
2022-06-21T16:09:21.857 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6436acc4b51b52635f8fa0e56cd79ba66c028d81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2022-06-21T16:09:21.858 INFO:scrape:3 jobs: ['6889282', '6889279', '6889287']
2022-06-21T16:09:21.858 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/pjd}}']
2022-06-21T16:09:21.859 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/no', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-no-extra-options}', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/suites/pjd}}', 'wsync/no}', 'wsync/yes}']
2022-06-21T16:09:21.859 INFO:scrape:
2022-06-21T16:09:21.860 INFO:scrape:Failure: {'smithi003.front.sepia.ceph.com': {'changed': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi003.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, 'smithi121.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'module_stderr': "Warning: Permanently added 'smithi121.front.sepia.ceph.com,172.21.15.121' (ECDSA) to the list of known hosts.\r\nControlSocket /home/teuthworker/.ansible/cp/a7df4ec898 already exists, disabling multiplexing\r\nConnection to smithi121.front.sepia.ceph.com closed.\r\n", 'module_stdout': "/usr/bin/python: can't open file '/home/ubuntu/.ansible/tmp/ansible-tmp-1655814164.51473-28061-57049058558779/AnsiballZ_apt.py': [Errno 2] No such file or directory\r\n", 'msg': 'MODULE FAILURE\nSee stdout/stderr for the exact error', 'rc': 2}}
2022-06-21T16:09:21.860 INFO:scrape:1 jobs: ['6889283']
2022-06-21T16:09:21.860 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/1', 'rhel_8}', 'scrub/no', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated}', 'tasks/{0-check-counter', 'workunit/suites/pjd}}', 'wsync/yes}']
2022-06-21T16:09:21.861 INFO:scrape:
2022-06-21T16:09:21.861 INFO:scrape:Failure: "2022-06-21T12:43:23.244046+0000 mon.a (mon.0) 488 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022-06-21T16:09:21.862 INFO:scrape:1 jobs: ['6889278']
2022-06-21T16:09:21.862 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snap-schedule}']
2022-06-21T16:09:21.863 INFO:scrape:
2022-06-21T16:09:21.863 INFO:scrape:Failure: "2022-06-21T13:04:07.739192+0000 mds.e (mds.0) 24 : cluster [WRN] client.4780 isn't responding to mclientcaps(revoke), ino 0x1000000408a pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.127744 seconds ago" in cluster log
2022-06-21T16:09:21.864 INFO:scrape:2 jobs: ['6889286', '6889288']
2022-06-21T16:09:21.864 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-06-21T16:09:21.865 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-06-21T16:09:21.865 INFO:scrape:
2022-06-21T16:09:21.865 INFO:scrape:Failure: Command failed on smithi003 with status 2: 'sudo dpkg -i /tmp/linux-image.deb'
2022-06-21T16:09:21.866 INFO:scrape:1 jobs: ['6889284']
2022-06-21T16:09:21.866 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}']
2022-06-21T16:09:21.867 INFO:scrape: