2023-11-09T16:47:01.573 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T16:47:01.574 INFO:teuthology.results:seed: '2387' 2023-11-09T16:48:20.750 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T16:48:20.754 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-09T17:41:24.400 INFO:teuthology.results:Tests finished! gathering results... 2023-11-09T17:41:25.055 INFO:scrape:Found 10 jobs 2023-11-09T17:41:25.854 INFO:scrape:Found 2 distinct failure reasons 2023-11-09T17:41:25.854 INFO:scrape:Failure: Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 pull' 2023-11-09T17:41:25.855 INFO:scrape:4 jobs: ['7452991', '7452990', '7452988', '7452989'] 2023-11-09T17:41:25.855 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2023-11-09T17:41:25.856 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-11-09T17:41:25.856 INFO:scrape: 2023-11-09T17:41:25.856 INFO:scrape:Timeout 120 running ceph --cluster ceph tell mds.1:0 scrub status 2023-11-09T17:41:25.857 INFO:scrape:1 jobs: ['7452984'] 2023-11-09T17:41:25.857 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-11-09T17:41:25.858 INFO:scrape: 2023-11-09T17:41:25.859 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7452984] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:44:06 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452984/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452984/ error during scrub thrashing: Command failed on smithi077 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [7452988] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:18:21 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452988/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452988/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa8371230d0b475b925eec9dfc30fc6f Command failed on smithi130 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 pull' [7452989] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:19:09 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452989/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452989/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ae18825d7b64313a1c117eac1b68296 Command failed on smithi159 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 pull' [7452990] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:18:48 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452990/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e4d5567a3804b9a9bde4287f90caf6d Command failed on smithi158 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 pull' [7452991] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:19:02 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452991/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452991/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5ce1b7a000745d9973f14679d04ac3f Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:23776148d05fd5bbb0eae6dde0ebb81fe2ceda46 pull' Pass ================================================================= [7452985] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:47:06 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452985/ [7452986] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:44:15 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452986/ [7452987] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:38:28 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452987/ [7452992] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:45:40 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452992/ [7452993] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:44:43 info: https://pulpito.ceph.com/mchangir-2023-11-09_16:46:30-fs:workload-main-distro-default-smithi/7452993/