2023-06-28T12:03:53.854 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T12:03:53.855 INFO:teuthology.results:seed: '7345' 2023-06-28T12:29:31.997 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T12:29:32.031 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-06-28T13:41:42.567 INFO:teuthology.results:Tests finished! gathering results... 2023-06-28T13:41:44.923 INFO:scrape:Found 32 jobs 2023-06-28T13:41:44.924 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319747/teuthology.log 2023-06-28T13:41:44.970 INFO:scrape:Found 2 distinct failure reasons 2023-06-28T13:41:44.971 INFO:scrape:Dead: None 2023-06-28T13:41:44.971 INFO:scrape:18 jobs: ['7319747', '7319746', '7319770', '7319744', '7319753', '7319755', '7319742', '7319745', '7319749', '7319748', '7319771', '7319743', '7319764', '7319750', '7319761', '7319766', '7319762', '7319769'] 2023-06-28T13:41:44.971 INFO:scrape: 2023-06-28T13:41:44.972 INFO:scrape:Failure: {'smithi067.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} 2023-06-28T13:41:44.972 INFO:scrape:1 jobs: ['7319751'] 2023-06-28T13:41:44.972 INFO:scrape:suites: ['debug/mgr', 'mgr_ttl_cache/enable', 'mon_election/classic', 'rados:mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'supported-random-distro$/{rhel_8}', 'tasks/module_selftest}'] 2023-06-28T13:41:44.973 INFO:scrape: 2023-06-28T13:41:44.974 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/ failed: 0 dead: 19 running: 0 waiting: 0 queued: 0 passed: 13 Dead ================================================================= [7319742] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/insights} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319742/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319742/ Error reimaging machines: [Errno 104] Connection reset by peer [7319743] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{centos_8} tasks/module_selftest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319743/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319743/ Error reimaging machines: [Errno 104] Connection reset by peer [7319744] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319744/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319744/ Error reimaging machines: [Errno 104] Connection reset by peer [7319745] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{rhel_8} tasks/progress} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319745/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319745/ Error reimaging machines: Failed to power on smithi067 [7319746] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_20.04} tasks/prometheus} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319746/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319746/ Error reimaging machines: [Errno 104] Connection reset by peer [7319747] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{rhel_8} tasks/workunits} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319747/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319747/ Error reimaging machines: [Errno 104] Connection reset by peer [7319748] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{centos_8} tasks/crash} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319748/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319748/ Error reimaging machines: [Errno 104] Connection reset by peer [7319749] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_8} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319749/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319749/ Error reimaging machines: Failed to power on smithi160 [7319750] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_8} tasks/insights} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319750/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319750/ Error reimaging machines: Expected smithi171's OS to be centos 8 but found rhel 8.6 [7319751] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{rhel_8} tasks/module_selftest} ----------------------------------------------------------------- time: 00:08:51 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319751/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319751/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d761bcfec9364bdf98211e490c176919 {'smithi067.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'err': " /dev/vg_nvme: already exists in filesystem\n Run `vgcreate --help' for more information.\n", 'failed': True, 'invocation': {'module_args': {'force': False, 'pesize': '4', 'pv_options': '', 'pvresize': False, 'pvs': ['/dev/nvme0n1'], 'state': 'present', 'vg': 'vg_nvme', 'vg_options': ''}}, 'item': {'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}, 'msg': "Creating volume group 'vg_nvme' failed", 'rc': 3}]}} [7319753] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{rhel_8} tasks/progress} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319753/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319753/ Error reimaging machines: [Errno 104] Connection reset by peer [7319755] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_8} tasks/workunits} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319755/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319755/ Error reimaging machines: [Errno 104] Connection reset by peer [7319761] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_8} tasks/progress} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319761/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319761/ Error reimaging machines: [Errno 104] Connection reset by peer [7319762] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/prometheus} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319762/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319762/ Error reimaging machines: [Errno 104] Connection reset by peer [7319764] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_20.04} tasks/crash} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319764/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319764/ Error reimaging machines: [Errno 104] Connection reset by peer [7319766] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{ubuntu_20.04} tasks/insights} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319766/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319766/ Error reimaging machines: Failed to power on smithi193 [7319769] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{rhel_8} tasks/progress} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319769/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319769/ Error reimaging machines: [Errno 104] Connection reset by peer [7319770] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_20.04} tasks/prometheus} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319770/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319770/ Error reimaging machines: [Errno 104] Connection reset by peer [7319771] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_8} tasks/workunits} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319771/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319771/ Error reimaging machines: [Errno 104] Connection reset by peer Pass ================================================================= [7319740] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:14:25 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319740/ [7319741] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{ubuntu_20.04} tasks/failover} time: 00:15:43 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319741/ [7319752] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/per_module_finisher_stats} time: 00:12:57 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319752/ [7319754] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_20.04} tasks/prometheus} time: 00:13:07 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319754/ [7319756] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:14:45 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319756/ [7319757] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{rhel_8} tasks/failover} time: 00:26:50 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319757/ [7319758] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{rhel_8} tasks/insights} time: 00:24:09 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319758/ [7319759] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_8} tasks/module_selftest} time: 01:04:58 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319759/ [7319760] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/per_module_finisher_stats} time: 00:16:43 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319760/ [7319763] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{ubuntu_20.04} tasks/workunits} time: 00:11:40 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319763/ [7319765] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{rhel_8} tasks/failover} time: 00:23:41 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319765/ [7319767] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:29:21 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319767/ [7319768] rados:mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{rhel_8} tasks/per_module_finisher_stats} time: 00:18:14 info: http://pulpito.ceph.com/pdonnell-2023-06-28_00:26:54-rados:mgr-wip-pdonnell-testing-20230627.204243-distro-default-smithi/7319768/