2023-11-09T18:03:40.718 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T18:03:40.719 INFO:teuthology.results:seed: '5963' 2023-11-09T18:04:32.910 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-09T18:04:32.914 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-09T18:44:35.590 INFO:teuthology.results:Tests finished! gathering results... 2023-11-09T18:44:36.492 INFO:scrape:Found 10 jobs 2023-11-09T18:44:36.507 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453017/teuthology.log 2023-11-09T18:44:36.818 INFO:scrape:Found 7 distinct failure reasons 2023-11-09T18:44:36.819 INFO:scrape:Failure: SSH connection to smithi033 was lost: 'sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm' 2023-11-09T18:44:36.820 INFO:scrape:2 jobs: ['7453019', '7453024'] 2023-11-09T18:44:36.820 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/1', 'session_timeout}', 'standby-replay'] 2023-11-09T18:44:36.821 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}', 'wsync/yes}'] 2023-11-09T18:44:36.821 INFO:scrape: 2023-11-09T18:44:36.821 INFO:scrape:Dead: None 2023-11-09T18:44:36.822 INFO:scrape:3 jobs: ['7453017', '7453018', '7453021'] 2023-11-09T18:44:36.822 INFO:scrape: 2023-11-09T18:44:36.823 INFO:scrape:Failure: ['Failed to connect to the host via ssh: ssh: connect to host smithi081.front.sepia.ceph.com port 22: No route to host', 'Failed to connect to the host via ssh: ssh: connect to host smithi130.front.sepia.ceph.com port 22: No route to host', 'Failed to connect to the host via ssh: ssh: connect to host smithi159.front.sepia.ceph.com port 22: No route to host'] 2023-11-09T18:44:36.823 INFO:scrape:1 jobs: ['7453016'] 2023-11-09T18:44:36.824 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-11-09T18:44:36.824 INFO:scrape: 2023-11-09T18:44:36.824 INFO:scrape:Failure: 2023-11-09T18:44:36.825 INFO:scrape:1 jobs: ['7453020'] 2023-11-09T18:44:36.825 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}'] 2023-11-09T18:44:36.826 INFO:scrape: 2023-11-09T18:44:36.826 INFO:scrape:Failure: ['Failed to connect to the host via ssh: ssh: connect to host smithi007.front.sepia.ceph.com port 22: No route to host', 'Failed to connect to the host via ssh: ssh: connect to host smithi120.front.sepia.ceph.com port 22: No route to host', 'Failed to connect to the host via ssh: ssh: connect to host smithi188.front.sepia.ceph.com port 22: No route to host', 'ubuntu@smithi007.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).'] 2023-11-09T18:44:36.826 INFO:scrape:1 jobs: ['7453015'] 2023-11-09T18:44:36.827 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2023-11-09T18:44:36.827 INFO:scrape: 2023-11-09T18:44:36.828 INFO:scrape:Failure: [Errno None] Unable to connect to port 22 on 172.21.15.98 2023-11-09T18:44:36.828 INFO:scrape:1 jobs: ['7453022'] 2023-11-09T18:44:36.828 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2023-11-09T18:44:36.829 INFO:scrape: 2023-11-09T18:44:36.829 INFO:scrape:Failure: Failed to reconnect to smithi121 2023-11-09T18:44:36.830 INFO:scrape:1 jobs: ['7453023'] 2023-11-09T18:44:36.830 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/no}'] 2023-11-09T18:44:36.831 INFO:scrape: 2023-11-09T18:44:36.832 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/ failed: 4 dead: 6 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7453016] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:03:27 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453016/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453016/ sentry: https://sentry.ceph.com/organizations/ceph/?query=adda2a057a0a49659a12eda0d9858ae2 ['Failed to connect to the host via ssh: ssh: connect to host smithi081.front.sepia.ceph.com port 22: No route to host', 'Failed to connect to the host via ssh: ssh: connect to host smithi130.front.sepia.ceph.com port 22: No route to host', 'Failed to connect to the host via ssh: ssh: connect to host smithi159.front.sepia.ceph.com port 22: No route to host'] [7453020] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453020/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453020/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56028fb7ddba4fa98af9106ad452ca01 [7453022] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453022/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453022/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04de4d8833b943d3865dc2b9c5a12c82 [Errno None] Unable to connect to port 22 on 172.21.15.98 [7453023] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453023/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453023/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6346a9f4a804b7da1e1d682482c8360 Failed to reconnect to smithi121 Dead ================================================================= [7453015] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:34:09 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453015/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453015/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87ed4993337e47b58e41383af5fa8734 ['Failed to connect to the host via ssh: ssh: connect to host smithi007.front.sepia.ceph.com port 22: No route to host', 'Failed to connect to the host via ssh: ssh: connect to host smithi120.front.sepia.ceph.com port 22: No route to host', 'Failed to connect to the host via ssh: ssh: connect to host smithi188.front.sepia.ceph.com port 22: No route to host', 'ubuntu@smithi007.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).'] [7453017] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453017/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453017/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7453018] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453018/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453018/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7453019] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453019/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453019/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31a70ff81da14450b4a8e7a9a01b34ea SSH connection to smithi033 was lost: 'sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm' [7453021] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453021/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453021/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7453024] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453024/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-11-09_18:03:10-fs:workload-main-distro-default-smithi/7453024/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c3178451e534c1d83cdcaa0bd7093f3 SSH connection to smithi038 was lost: 'sudo yum install -y kernel'