2023-11-09T05:50:44.844 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-09T05:50:44.845 INFO:teuthology.results:seed: '571'
2023-11-09T06:15:55.170 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-09T06:15:55.173 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-09T18:16:46.043 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-11-09T18:16:46.929 INFO:scrape:Found 10 jobs
2023-11-09T18:16:48.621 INFO:scrape:Found 2 distinct failure reasons
2023-11-09T18:16:48.622 INFO:scrape:Dead: b'2023-11-09T18:07:23.697 DEBUG:teuthology.exit:Finished running handlers'
2023-11-09T18:16:48.622 INFO:scrape:1 jobs: ['7452581']
2023-11-09T18:16:48.623 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}']
2023-11-09T18:16:48.623 INFO:scrape:
2023-11-09T18:16:48.624 INFO:scrape:Timeout 120 running ceph --cluster ceph tell mds.1:0 scrub status
2023-11-09T18:16:48.624 INFO:scrape:2 jobs: ['7452580', '7452578']
2023-11-09T18:16:48.624 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'wsync/yes}']
2023-11-09T18:16:48.625 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2023-11-09T18:16:48.625 INFO:scrape:
2023-11-09T18:16:48.626 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/
failed:  2
dead:    0
running: 1
waiting: 0
queued:  0
passed:  7


Fail
=================================================================
[7452578]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:50:10
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452578/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452578/

    error during scrub thrashing: Command failed on smithi103 with status 124:
    'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
    timeout 120 ceph --cluster ceph tell mds.1:0 scrub status'

[7452580]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:47:04
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452580/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452580/

    error during scrub thrashing: Command failed on smithi050 with status 124:
    'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
    timeout 120 ceph --cluster ceph tell mds.1:0 scrub status'



Running
=================================================================
[7452581] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452581/



Pass
=================================================================
[7452579] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   00:46:43
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452579/

[7452582] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   00:43:22
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452582/

[7452583] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   00:42:48
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452583/

[7452584] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   00:43:15
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452584/

[7452585] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   00:35:36
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452585/

[7452586] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   00:49:00
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452586/

[7452587] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}}
time:   00:46:16
info:   https://pulpito.ceph.com/mchangir-2023-11-09_05:47:18-fs:workload-main-distro-default-smithi/7452587/