2022-09-09T00:17:58.651 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-09T00:17:58.653 INFO:teuthology.results:subset: '34/64' 2022-09-09T00:17:58.654 INFO:teuthology.results:seed: '6239' 2022-09-09T01:08:53.052 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-09T01:08:53.075 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-09-09T02:47:00.778 INFO:teuthology.results:Tests finished! gathering results... 2022-09-09T02:47:01.621 INFO:scrape:Found 10 jobs 2022-09-09T02:47:03.150 INFO:scrape:Found 2 distinct failure reasons 2022-09-09T02:47:03.152 INFO:scrape:Failure: 'Remote' object has no attribute '_CephFSMount__read_file' 2022-09-09T02:47:03.152 INFO:scrape:9 jobs: ['7021154', '7021152', '7021150', '7021155', '7021149', '7021156', '7021153', '7021148', '7021151'] 2022-09-09T02:47:03.153 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2022-09-09T02:47:03.154 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}', 'wsync/yes}'] 2022-09-09T02:47:03.155 INFO:scrape: 2022-09-09T02:47:03.155 INFO:scrape:Failure: Command failed on smithi109 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33dface8-2fde-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi109:vg_nvme/lv_4' 2022-09-09T02:47:03.156 INFO:scrape:1 jobs: ['7021157'] 2022-09-09T02:47:03.157 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2022-09-09T02:47:03.158 INFO:scrape: 2022-09-09T02:47:03.160 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/ failed: 2 dead: 8 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7021151] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:22:52 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021151/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021151/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f16bf6af3754647bbd19d727a72cfbe 'Remote' object has no attribute '_CephFSMount__read_file' [7021157] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:15:04 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021157/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021157/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ebaae2fdee5447d3a76651f12a6fb8b9 Command failed on smithi109 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:152194198413033811b31149de909a7ec5852a7f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33dface8-2fde-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi109:vg_nvme/lv_4' Dead ================================================================= [7021148] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:22:08 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021148/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021148/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d299fde4be74108ab184e8180529908 'Remote' object has no attribute '_CephFSMount__read_file' [7021149] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:22:42 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021149/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021149/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55a513bd7b8548898f17fd891976b194 'Remote' object has no attribute '_CephFSMount__read_file' [7021150] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:28:40 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021150/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021150/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae00362b65884f5eba94ef1893be5893 'Remote' object has no attribute '_CephFSMount__read_file' [7021152] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:21:42 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021152/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021152/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01919c6538b945baaa2068676cc69e9b 'Remote' object has no attribute '_CephFSMount__read_file' [7021153] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:23:57 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021153/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021153/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d006b76cf51849fc9916a02c101b6f4c 'Remote' object has no attribute '_CephFSMount__read_file' [7021154] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:28:16 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021154/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021154/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9941d24d34ed4a608a77abdf7197c1ad 'Remote' object has no attribute '_CephFSMount__read_file' [7021155] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:23:44 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021155/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021155/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4408dad6ca8f423cbb9522e8db29bac7 'Remote' object has no attribute '_CephFSMount__read_file' [7021156] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:22:15 info: http://pulpito.ceph.com/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021156/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-09_00:11:26-fs:workload-wip-pdonnell-testing-20220907.015905-distro-default-smithi/7021156/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de5c6aa8965c462db64b318f9a2d80d4 'Remote' object has no attribute '_CephFSMount__read_file'