2023-03-21T15:13:48.670 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3 2023-03-21T15:13:48.671 INFO:teuthology.results:subset: '17/64' 2023-03-21T15:13:48.671 INFO:teuthology.results:seed: '7997' 2023-03-21T15:30:47.046 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3 2023-03-21T15:30:47.072 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-22T03:31:42.549 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-03-22T03:31:43.405 INFO:scrape:Found 10 jobs 2023-03-22T03:31:43.409 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215396/teuthology.log 2023-03-22T03:31:44.444 INFO:scrape:Found 3 distinct failure reasons 2023-03-22T03:31:44.445 INFO:scrape:Dead: None 2023-03-22T03:31:44.446 INFO:scrape:1 jobs: ['7215396'] 2023-03-22T03:31:44.446 INFO:scrape: 2023-03-22T03:31:44.446 INFO:scrape:Failure: machine smithi063.front.sepia.ceph.com is locked by scheduled_yuriw@teuthology, not scheduled_pdonnell@teuthology 2023-03-22T03:31:44.447 INFO:scrape:1 jobs: ['7215395'] 2023-03-22T03:31:44.447 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/direct_io}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-03-22T03:31:44.448 INFO:scrape: 2023-03-22T03:31:44.448 INFO:scrape:Dead: b'2023-03-22T03:23:41.124 DEBUG:teuthology.exit:Finished running handlers' 2023-03-22T03:31:44.449 INFO:scrape:1 jobs: ['7215389'] 2023-03-22T03:31:44.449 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-03-22T03:31:44.450 INFO:scrape: 2023-03-22T03:31:44.451 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/ failed: 1 dead: 1 running: 1 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7215395] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215395/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d7f80ba1ccb49bf8d3dc40f001e979e machine smithi063.front.sepia.ceph.com is locked by scheduled_yuriw@teuthology, not scheduled_pdonnell@teuthology Dead ================================================================= [7215396] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215396/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215396/ Error reimaging machines: Failed to power on smithi063 Running ================================================================= [7215389] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215389/ Pass ================================================================= [7215388] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 01:44:50 info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215388/ [7215390] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} time: 01:12:44 info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215390/ [7215391] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 01:23:01 info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215391/ [7215392] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} time: 02:08:03 info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215392/ [7215393] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 01:06:26 info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215393/ [7215394] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:37:54 info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215394/ [7215397] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} time: 01:05:21 info: http://pulpito.ceph.com/pdonnell-2023-03-21_15:13:06-fs:workload-wip-pdonnell-testing-20230316.214637-distro-default-smithi/7215397/