2023-03-16T16:10:39.275 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-16T16:10:41.077 INFO:teuthology.results:subset: '40/128' 2023-03-16T16:10:41.077 INFO:teuthology.results:seed: '1233' 2023-03-16T16:23:22.953 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-16T16:23:24.397 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-16T19:53:41.614 INFO:teuthology.results:Tests finished! gathering results... 2023-03-16T19:53:42.769 INFO:scrape:Found 6 jobs 2023-03-16T19:53:48.157 INFO:scrape:Found 2 distinct failure reasons 2023-03-16T19:53:53.853 INFO:scrape:Failure: SSH connection to smithi019 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2023-03-16T19:53:54.521 INFO:scrape:4 jobs: ['7210222', '7210225', '7210224', '7210221'] 2023-03-16T19:53:54.522 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'session_timeout}', 'standby-replay', 'wsync/no}'] 2023-03-16T19:53:54.522 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/norstats}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-03-16T19:53:54.522 INFO:scrape: 2023-03-16T19:53:54.522 INFO:scrape:Failure: Command failed on smithi072 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8e902cc-c418-11ed-9afb-001a4aab830c --force' 2023-03-16T19:53:56.089 INFO:scrape:1 jobs: ['7210220'] 2023-03-16T19:53:56.089 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}'] 2023-03-16T19:53:56.089 INFO:scrape: 2023-03-16T19:53:56.090 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7210220] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 03:28:30 info: http://pulpito.ceph.com/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210220/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210220/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aed4ed29e325470fbddfe07a5172533f Command failed on smithi072 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a8e902cc-c418-11ed-9afb-001a4aab830c --force' [7210221] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 03:20:05 info: http://pulpito.ceph.com/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210221/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210221/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d6dcc65dacc40bebb6f4d8023371944 SSH connection to smithi037 was lost: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 77e7049c-c41a-11ed-9afb-001a4aab830c --force' [7210222] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 03:16:33 info: http://pulpito.ceph.com/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210222/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210222/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85ea575ada4744a2a709f3306cb51d72 SSH connection to smithi019 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' [7210224] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 03:14:39 info: http://pulpito.ceph.com/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210224/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210224/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8921c337f0b4ccc9acec26821f47e58 SSH connection to smithi040 was lost: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f585e7ec-c41a-11ed-9afb-001a4aab830c --force' [7210225] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 03:13:02 info: http://pulpito.ceph.com/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210225/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210225/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fa0767da4b945e0a8a0a40106c93335 SSH connection to smithi050 was lost: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 593e2d4e-c41b-11ed-9afb-001a4aab830c --force' Pass ================================================================= [7210223] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 01:14:51 info: http://pulpito.ceph.com/pdonnell-2023-03-16_16:09:29-fs-wip-pdonnell-testing-20230316.143428-distro-default-smithi/7210223/