2024-02-13T09:50:11.889 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-13T09:50:11.889 INFO:teuthology.results:seed: '4351' 2024-02-13T10:03:38.399 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-13T10:03:38.403 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-13T11:56:47.132 INFO:teuthology.results:Tests finished! gathering results... 2024-02-13T11:56:48.221 INFO:scrape:Found 15 jobs 2024-02-13T11:56:48.222 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557874/teuthology.log 2024-02-13T11:56:48.412 INFO:scrape:Found 3 distinct failure reasons 2024-02-13T11:56:48.413 INFO:scrape:Dead: None 2024-02-13T11:56:48.413 INFO:scrape:1 jobs: ['7557874'] 2024-02-13T11:56:48.413 INFO:scrape: 2024-02-13T11:56:48.413 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_info_if_clone_pending_for_no_group (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2024-02-13T11:56:48.413 INFO:scrape:2 jobs: ['7557875', '7557870'] 2024-02-13T11:56:48.413 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2024-02-13T11:56:48.413 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}'] 2024-02-13T11:56:48.414 INFO:scrape: 2024-02-13T11:56:48.414 INFO:scrape:Failure: local variable 'version' referenced before assignment 2024-02-13T11:56:48.414 INFO:scrape:2 jobs: ['7557883', '7557880'] 2024-02-13T11:56:48.414 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides'] 2024-02-13T11:56:48.414 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs:volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/clone}}', 'test/snapshot}}'] 2024-02-13T11:56:48.414 INFO:scrape: 2024-02-13T11:56:48.415 INFO:teuthology.results:Sending results to neesingh@redhat.com: Test Run: neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/ failed: 4 dead: 1 running: 0 waiting: 0 queued: 0 passed: 10 Fail ================================================================= [7557870] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:50:28 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557870/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557870/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b293f0d8eb1d402c8a2b6a127cfa9f1b Test failure: test_subvolume_snapshot_info_if_clone_pending_for_no_group (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7557875] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:57:02 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557875/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557875/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9d16939b2dd482ea749a9981d58e393 Test failure: test_subvolume_snapshot_info_if_clone_pending_for_no_group (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [7557880] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557880/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557880/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94863f7798544500bf88260f1f5f37de local variable 'version' referenced before assignment [7557883] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557883/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557883/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eba24971800946738d6279447da4eac8 local variable 'version' referenced before assignment Dead ================================================================= [7557874] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557874/ log: http://qa-proxy.ceph.com/teuthology/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557874/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7557869] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:54:47 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557869/ [7557871] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:14:21 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557871/ [7557872] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:18:01 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557872/ [7557873] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:27:36 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557873/ [7557876] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:08:56 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557876/ [7557877] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:21:22 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557877/ [7557878] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:36:41 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557878/ [7557879] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:37:26 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557879/ [7557881] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:08:59 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557881/ [7557882] fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:16:52 info: https://pulpito.ceph.com/neesingh-2024-02-13_09:49:19-fs:volumes-wip-neesingh-testing1-20240208-distro-default-smithi/7557882/