2023-10-04T23:32:46.882 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-04T23:32:46.882 INFO:teuthology.results:subset: '435/512' 2023-10-04T23:32:46.882 INFO:teuthology.results:seed: '3588' 2023-10-05T01:59:15.116 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-05T01:59:15.142 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-05T11:53:02.472 INFO:teuthology.results:Tests finished! gathering results... 2023-10-05T11:53:09.236 INFO:scrape:Found 75 jobs 2023-10-05T11:53:10.186 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:10.187 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:10.187 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:10.187 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.443 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.443 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.443 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.443 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.487 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.488 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.488 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.488 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.501 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.501 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.502 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.502 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.506 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.506 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.506 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.506 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.546 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.546 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.546 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.546 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.624 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.624 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.624 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.624 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.632 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.632 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.633 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.633 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.644 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.644 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.645 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.645 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.649 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.649 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.649 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.649 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.653 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.653 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.653 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.654 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.658 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.658 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.658 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.658 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.662 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.662 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.662 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.662 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.739 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.739 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.740 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.740 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.744 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.744 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.744 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.745 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.759 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.759 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.760 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.760 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.764 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.764 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.764 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.764 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.768 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.769 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.769 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.769 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.773 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.773 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.773 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.774 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.780 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.780 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.780 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.780 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.784 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.784 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.785 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:12.785 WARNING:scrape:Saw end of BT but not start 2023-10-05T11:53:15.567 INFO:scrape:Found 4 distinct failure reasons 2023-10-05T11:53:15.568 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-10-05T11:53:15.568 INFO:scrape:9 jobs: ['7410758', '7410738', '7410730', '7410760', '7410695', '7410708', '7410740', '7410712', '7410716'] 2023-10-05T11:53:15.568 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-10-05T11:53:15.568 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/postgres}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'delegate_inos/default', 'delegate_inos/disabled', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2023-10-05T11:53:15.569 INFO:scrape: 2023-10-05T11:53:15.569 INFO:scrape:Dead: b'2023-10-05T11:42:58.715 DEBUG:teuthology.exit:Finished running handlers' 2023-10-05T11:53:15.569 INFO:scrape:1 jobs: ['7410694'] 2023-10-05T11:53:15.569 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'delegate_inos/default', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2023-10-05T11:53:15.569 INFO:scrape: 2023-10-05T11:53:15.569 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b2712a8330a69bb776c9f1eae25b24a3195735c9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-10-05T11:53:15.569 INFO:scrape:2 jobs: ['7410757', '7410697'] 2023-10-05T11:53:15.570 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'delegate_inos/disabled', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-10-05T11:53:15.570 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'delegate_inos/disabled', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-10-05T11:53:15.570 INFO:scrape: 2023-10-05T11:53:15.570 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi029 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b2712a8330a69bb776c9f1eae25b24a3195735c9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2023-10-05T11:53:15.570 INFO:scrape:1 jobs: ['7410739'] 2023-10-05T11:53:15.570 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'delegate_inos/disabled', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-10-05T11:53:15.571 INFO:scrape: 2023-10-05T11:53:15.575 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/ failed: 12 dead: 1 running: 0 waiting: 0 queued: 0 passed: 62 Fail ================================================================= [7410695] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:54:04 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410695/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410695/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7410697] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:53:52 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410697/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410697/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f11e606e19548bf83c0738f1c1d8051 Command failed (workunit test suites/dbench.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b2712a8330a69bb776c9f1eae25b24a3195735c9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7410708] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:48:06 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410708/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410708/ error during scrub thrashing: rank damage found: {'backtrace'} [7410712] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:07:51 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410712/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410712/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7410716] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:44:11 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410716/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410716/ error during scrub thrashing: rank damage found: {'backtrace'} [7410730] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:37:20 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410730/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410730/ error during scrub thrashing: rank damage found: {'backtrace'} [7410738] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:47:10 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410738/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410738/ error during scrub thrashing: rank damage found: {'backtrace'} [7410739] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:43:43 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410739/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410739/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a21d965b5b994c6bb9a309c26ff87922 Command failed (workunit test kernel_untar_build.sh) on smithi029 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b2712a8330a69bb776c9f1eae25b24a3195735c9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [7410740] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:58:23 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410740/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410740/ error during scrub thrashing: rank damage found: {'backtrace'} [7410757] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:40:32 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410757/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410757/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56d8696bf6294111b3b389d708dee7df Command failed (workunit test suites/dbench.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b2712a8330a69bb776c9f1eae25b24a3195735c9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7410758] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:42:41 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410758/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410758/ error during scrub thrashing: rank damage found: {'backtrace'} [7410760] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:39:55 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410760/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410760/ error during scrub thrashing: rank damage found: {'backtrace'} Dead ================================================================= [7410694] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410694/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410694/ hit max job timeout Pass ================================================================= [7410692] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:39:27 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410692/ [7410693] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:40:31 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410693/ [7410696] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:41:04 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410696/ [7410698] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:43:27 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410698/ [7410699] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:40:50 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410699/ [7410700] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:36:03 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410700/ [7410701] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:42:03 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410701/ [7410702] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:27:40 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410702/ [7410703] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:41:05 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410703/ [7410704] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:48:13 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410704/ [7410705] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:40:41 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410705/ [7410706] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:28:22 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410706/ [7410707] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:31:26 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410707/ [7410709] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 00:47:59 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410709/ [7410710] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 00:52:49 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410710/ [7410711] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:44:03 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410711/ [7410713] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:40:13 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410713/ [7410714] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:37:28 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410714/ [7410715] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:32:15 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410715/ [7410717] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:32:00 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410717/ [7410718] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:36:54 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410718/ [7410719] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 00:40:58 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410719/ [7410720] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:42:19 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410720/ [7410721] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} time: 00:31:59 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410721/ [7410722] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:36:06 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410722/ [7410723] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}} time: 00:46:51 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410723/ [7410724] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} time: 03:12:32 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410724/ [7410725] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} time: 00:46:31 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410725/ [7410726] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} time: 00:52:37 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410726/ [7410727] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} time: 01:10:24 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410727/ [7410728] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} time: 01:05:20 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410728/ [7410729] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} time: 00:32:04 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410729/ [7410731] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} time: 00:36:33 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410731/ [7410732] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:42:53 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410732/ [7410733] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:41:08 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410733/ [7410734] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:52:19 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410734/ [7410735] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:33:11 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410735/ [7410736] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} time: 00:44:06 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410736/ [7410737] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:41:12 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410737/ [7410741] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:42:49 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410741/ [7410742] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:05:07 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410742/ [7410743] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 00:54:14 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410743/ [7410744] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:35:57 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410744/ [7410745] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:40:53 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410745/ [7410746] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:35:52 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410746/ [7410747] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:38:41 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410747/ [7410748] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:42:25 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410748/ [7410749] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:41:16 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410749/ [7410750] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:27:40 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410750/ [7410751] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:38:43 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410751/ [7410752] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:38:43 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410752/ [7410753] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 00:58:40 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410753/ [7410754] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} time: 00:48:00 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410754/ [7410755] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:40:33 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410755/ [7410756] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:45:18 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410756/ [7410759] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:41:34 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410759/ [7410761] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:41:35 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410761/ [7410762] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:28:32 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410762/ [7410763] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:43:32 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410763/ [7410764] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:49:51 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410764/ [7410765] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/disabled mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:42:24 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410765/ [7410766] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} delegate_inos/default mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:28:36 info: http://pulpito.ceph.com/vshankar-2023-10-04_17:07:09-fs:workload-wip-61009-wo-testing-default-smithi/7410766/