2022-07-21T20:19:56.636 INFO:root:teuthology version: 1.1.1.dev389+g8d598431 2022-07-21T20:19:56.637 INFO:teuthology.results:subset: '11/32' 2022-07-21T20:19:56.637 INFO:teuthology.results:seed: '5381' 2022-07-21T20:19:59.422 INFO:root:teuthology version: 1.1.1.dev389+g8d598431 2022-07-21T20:19:59.423 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-21T20:56:02.931 INFO:teuthology.results:Tests finished! gathering results... 2022-07-21T20:56:03.311 INFO:scrape:Found 6 jobs 2022-07-21T20:56:03.312 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943055/teuthology.log 2022-07-21T20:56:06.835 INFO:scrape:Found 5 distinct failure reasons 2022-07-21T20:56:06.836 INFO:scrape:Dead: None 2022-07-21T20:56:06.836 INFO:scrape:1 jobs: ['6943055'] 2022-07-21T20:56:06.837 INFO:scrape: 2022-07-21T20:56:06.837 INFO:scrape:Failure: SSH connection to smithi083 was lost: 'sudo yum upgrade -y linux-firmware' 2022-07-21T20:56:06.838 INFO:scrape:1 jobs: ['6943056'] 2022-07-21T20:56:06.838 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:volumes/{begin', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-21T20:56:06.839 INFO:scrape: 2022-07-21T20:56:06.840 INFO:scrape:Failure: Test failure: test_subvolume_group_create_idempotence_data_pool (tasks.cephfs.test_volumes.TestSubvolumeGroups) 2022-07-21T20:56:06.841 INFO:scrape:2 jobs: ['6943054', '6943059'] 2022-07-21T20:56:06.841 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-21T20:56:06.842 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs:volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-21T20:56:06.842 INFO:scrape: 2022-07-21T20:56:06.843 INFO:scrape:Failure: Test failure: test_subvolume_snapshot_metadata_after_snapshot_remove (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) 2022-07-21T20:56:06.843 INFO:scrape:1 jobs: ['6943057'] 2022-07-21T20:56:06.844 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:volumes/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/volumes/{overrides', 'test/snapshot}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-21T20:56:06.853 INFO:scrape: 2022-07-21T20:56:06.854 INFO:scrape:Failure: Command failed on smithi109 with status 1: 'sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm' 2022-07-21T20:56:06.857 INFO:scrape:1 jobs: ['6943058'] 2022-07-21T20:56:06.858 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{no_client_pidfile', 'tasks/volume-client}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-21T20:56:06.858 INFO:scrape: 2022-07-21T20:56:06.859 INFO:teuthology.results:Sending results to dgallowa@redhat.com: Test Run: dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/ failed: 4 dead: 2 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [6943054] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:25:04 info: http://pulpito.ceph.com/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943054/ log: http://qa-proxy.ceph.com/teuthology/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943054/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d71c467b78924176ab678232fc120ccf Test failure: test_subvolume_group_create_idempotence_data_pool (tasks.cephfs.test_volumes.TestSubvolumeGroups) [6943057] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:22:46 info: http://pulpito.ceph.com/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943057/ log: http://qa-proxy.ceph.com/teuthology/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943057/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eb8c2578dbb9408a95179e803dfa146a Test failure: test_subvolume_snapshot_metadata_after_snapshot_remove (tasks.cephfs.test_volumes.TestSubvolumeSnapshots) [6943058] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volume-client} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943058/ log: http://qa-proxy.ceph.com/teuthology/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943058/ sentry: https://sentry.ceph.com/organizations/ceph/?query=26d513b51d7b45bcbcbe0eddd2285001 Command failed on smithi109 with status 1: 'sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm' [6943059] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:22:17 info: http://pulpito.ceph.com/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943059/ log: http://qa-proxy.ceph.com/teuthology/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943059/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffe5e59b45c04ac88c47bb1ae6df8917 Test failure: test_subvolume_group_create_idempotence_data_pool (tasks.cephfs.test_volumes.TestSubvolumeGroups) Dead ================================================================= [6943055] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943055/ log: http://qa-proxy.ceph.com/teuthology/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943055/ Error reimaging machines: 'ssh_keyscan smithi083.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds [6943056] fs:volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943056/ log: http://qa-proxy.ceph.com/teuthology/dgalloway-2022-07-21_20:04:54-fs:volumes-pacific-testing-default-smithi/6943056/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19cdf5e835ff469e833494a8ceac61ad SSH connection to smithi083 was lost: 'sudo yum upgrade -y linux-firmware'