2023-03-21T22:57:48.092 INFO:root:teuthology version: 0.0.1.dev66+g73b89aa 2023-03-21T22:57:48.093 INFO:teuthology.results:subset: '12/64' 2023-03-21T22:57:48.094 INFO:teuthology.results:seed: '5498' 2023-03-21T23:18:26.713 INFO:root:teuthology version: 0.0.1.dev66+g73b89aa 2023-03-21T23:18:26.716 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-22T11:19:23.697 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-03-22T11:19:24.578 INFO:scrape:Found 10 jobs 2023-03-22T11:19:29.695 INFO:scrape:Found 3 distinct failure reasons 2023-03-22T11:19:29.696 INFO:scrape:Failure: Command failed on smithi042 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" 2023-03-22T11:19:29.696 INFO:scrape:1 jobs: ['7215563'] 2023-03-22T11:19:29.697 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2023-03-22T11:19:29.697 INFO:scrape: 2023-03-22T11:19:29.697 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-03-22T11:19:29.698 INFO:scrape:4 jobs: ['7215568', '7215562', '7215565', '7215570'] 2023-03-22T11:19:29.698 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-03-22T11:19:29.698 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/iozone}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-03-22T11:19:29.699 INFO:scrape: 2023-03-22T11:19:29.699 INFO:scrape:Dead: b'2023-03-22T11:19:27.678 INFO:journalctl@ceph.mon.c.smithi187.stdout:Mar 22 11:19:27 smithi187 ceph-mon[94501]: pgmap v22705: 129 pgs: 129 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.5 KiB/s wr, 7 op/s' 2023-03-22T11:19:29.699 INFO:scrape:1 jobs: ['7215571'] 2023-03-22T11:19:29.700 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-03-22T11:19:29.700 INFO:scrape: 2023-03-22T11:19:29.701 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/ failed: 5 dead: 0 running: 1 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [7215562] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:41:25 info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215562/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215562/ error during scrub thrashing: rank damage found: {'backtrace'} [7215563] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:40:35 info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215563/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215563/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83543aa603b84325a06eaa7d4091ecbe Command failed on smithi042 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i'" [7215565] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:34:50 info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215565/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215565/ error during scrub thrashing: rank damage found: {'backtrace'} [7215568] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:35:42 info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215568/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215568/ error during scrub thrashing: rank damage found: {'backtrace'} [7215570] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:57:28 info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215570/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215570/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds Running ================================================================= [7215571] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215571/ Pass ================================================================= [7215564] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} time: 01:13:45 info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215564/ [7215566] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} time: 00:42:37 info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215566/ [7215567] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:34:16 info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215567/ [7215569] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:34:57 info: http://pulpito.ceph.com/pdonnell-2023-03-21_22:57:01-fs:workload-wip-pdonnell-testing-20230321.204343-distro-default-smithi/7215569/