2023-08-16T11:28:38.362 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-16T11:28:38.363 INFO:teuthology.results:subset: '486/1024' 2023-08-16T11:28:38.364 INFO:teuthology.results:seed: '8976' 2023-08-16T11:34:13.559 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-16T11:34:13.564 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-16T23:35:08.850 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-08-16T23:35:09.470 INFO:scrape:Found 9 jobs 2023-08-16T23:35:10.289 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369841/teuthology.log 2023-08-16T23:35:10.965 INFO:scrape:Found 4 distinct failure reasons 2023-08-16T23:35:10.966 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-08-16T23:35:10.966 INFO:scrape:2 jobs: ['7369835', '7369843'] 2023-08-16T23:35:10.967 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-08-16T23:35:10.967 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}'] 2023-08-16T23:35:10.967 INFO:scrape: 2023-08-16T23:35:10.968 INFO:scrape:Dead: None 2023-08-16T23:35:10.968 INFO:scrape:1 jobs: ['7369841'] 2023-08-16T23:35:10.968 INFO:scrape: 2023-08-16T23:35:10.969 INFO:scrape:Dead: b'2023-08-16T23:35:02.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-08-16T23:35:10.969 INFO:scrape:4 jobs: ['7369838', '7369836', '7369840', '7369842'] 2023-08-16T23:35:10.969 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-08-16T23:35:10.970 INFO:scrape:suites union: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'rhel_8}', 'tasks/multifs-auth}', 'tasks/{0-client'] 2023-08-16T23:35:10.970 INFO:scrape: 2023-08-16T23:35:10.970 INFO:scrape:Failure: No module named 'tasks.fs' 2023-08-16T23:35:10.970 INFO:scrape:1 jobs: ['7369839'] 2023-08-16T23:35:10.971 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/suites/iozone}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2023-08-16T23:35:10.971 INFO:scrape: 2023-08-16T23:35:10.972 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/ failed: 3 dead: 1 running: 4 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7369835] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 09:33:31 info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369835/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369835/ sentry: https://sentry.ceph.com/organizations/ceph/?query=514b0a9fb6fc401cbfd79cecf1cf01e9 Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7369839] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:28:02 info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369839/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369839/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6207ca44e734a979ea1cceb8269e204 No module named 'tasks.fs' [7369843] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 08:11:20 info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369843/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369843/ sentry: https://sentry.ceph.com/organizations/ceph/?query=095f17713ff44845a247ae9be199b79b Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) Dead ================================================================= [7369841] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369841/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369841/ Error reimaging machines: Timeout opening channel. Running ================================================================= [7369836] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369836/ [7369838] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}} info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369838/ [7369840] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369840/ [7369842] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369842/ Pass ================================================================= [7369837] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:15:42 info: http://pulpito.ceph.com/vshankar-2023-08-16_11:14:57-fs-wip-vshankar-testing-20230809.035933-testing-default-smithi/7369837/