2023-11-09T18:36:13.689 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T18:36:13.690 INFO:teuthology.results:seed: '9563' 2023-11-09T18:37:04.217 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T18:37:04.221 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-09T19:15:07.187 INFO:teuthology.results:Tests finished! gathering results... 2023-11-09T19:15:07.652 INFO:scrape:Found 10 jobs 2023-11-09T19:15:08.652 INFO:scrape:Found 2 distinct failure reasons 2023-11-09T19:15:08.653 INFO:scrape:Failure: name 'pre' is not defined 2023-11-09T19:15:08.653 INFO:scrape:9 jobs: ['7453030', '7453032', '7453033', '7453029', '7453031', '7453034', '7453036', '7453035', '7453028'] 2023-11-09T19:15:08.654 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/1', 'session_timeout}', 'standby-replay'] 2023-11-09T19:15:08.654 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2023-11-09T19:15:08.655 INFO:scrape: 2023-11-09T19:15:08.655 INFO:scrape:Failure: Command failed on smithi007 with status 1: 'set -ex\nsudo dd of=/home/ubuntu/cephtest/valgrind.supp' 2023-11-09T19:15:08.655 INFO:scrape:1 jobs: ['7453027'] 2023-11-09T19:15:08.656 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-11-09T19:15:08.656 INFO:scrape: 2023-11-09T19:15:08.657 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/ failed: 10 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7453027] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:12:35 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453027/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453027/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cf9df51324ad46a795e9f416342154f7 Command failed on smithi007 with status 1: 'set -ex\nsudo dd of=/home/ubuntu/cephtest/valgrind.supp' [7453028] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:23:31 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453028/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453028/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2f527ac3d1042e395fc48b0804bb567 name 'pre' is not defined [7453029] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:22:53 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453029/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453029/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae98bddef3b04296a79217f1bac9d8fe name 'pre' is not defined [7453030] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:22:58 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453030/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453030/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bcb3e885defa450fae728166cbec4a6a name 'pre' is not defined [7453031] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:27:10 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453031/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453031/ sentry: https://sentry.ceph.com/organizations/ceph/?query=74836a9e2e24410f8534bb05ff65b1ce name 'pre' is not defined [7453032] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:27:11 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453032/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453032/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31823bd084fe495f89e478a9c8271560 name 'pre' is not defined [7453033] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:27:42 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453033/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453033/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c8b1b98d1ea40c88b0271822f873ddb name 'pre' is not defined [7453034] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:27:04 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453034/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453034/ sentry: https://sentry.ceph.com/organizations/ceph/?query=793c9ade8a2f495d928ac430c17de11c name 'pre' is not defined [7453035] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:23:35 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453035/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453035/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1628911c638646a5ab41afd7b534568b name 'pre' is not defined [7453036] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:24:01 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453036/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:35:30-fs:workload-main-distro-default-smithi/7453036/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38127816fe114f5aa9c867ce84dc1b6f name 'pre' is not defined