2024-02-26T21:55:58.535 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-26T21:55:58.536 INFO:teuthology.results:subset: '111/120000' 2024-02-26T21:55:58.536 INFO:teuthology.results:seed: '7610' 2024-02-26T22:02:47.270 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-26T22:02:47.274 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-27T01:35:03.789 INFO:teuthology.results:Tests finished! gathering results... 2024-02-27T01:35:04.186 INFO:scrape:Found 5 jobs 2024-02-27T01:35:05.446 INFO:scrape:Found 2 distinct failure reasons 2024-02-27T01:35:05.447 INFO:scrape:Timeout 120 running ceph --cluster ceph health --format=json 2024-02-27T01:35:05.447 INFO:scrape:1 jobs: ['7574174'] 2024-02-27T01:35:05.447 INFO:scrape:suites: ['supported-all-distro/ubuntu_latest}', 'upgrade:reef-p2p/reef-p2p-parallel/{point-to-point-upgrade'] 2024-02-27T01:35:05.447 INFO:scrape: 2024-02-27T01:35:05.447 INFO:scrape:Crash: timeout expired in wait_until_healthy 2024-02-27T01:35:05.447 INFO:scrape:ceph version 18.2.1-638-g055f1d8d (055f1d8d2e5cbd7e37bfd1e7bd8c27a3b6e775e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f68921c0cf0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f689107009b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f689107653c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f6891076597] 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f68910767f8] 8: (ceph::buffer::v15_2_0::list::iterator_impl::copy(unsigned int, char*)+0xa5) [0x7f68937d5385] 9: (MDSMap::decode(ceph::buffer::v15_2_0::list::iterator_impl&)+0xa64) [0x7f68939f0eb4] 10: (MDSDaemon::handle_mds_map(boost::intrusive_ptr const&)+0x216) [0x564e66be7386] 11: (MDSDaemon::handle_core_message(boost::intrusive_ptr const&)+0x367) [0x564e66beae87] 12: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x177) [0x564e66beb607] 13: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6893689af8] 14: (DispatchQueue::entry()+0x50f) [0x7f6893686c9f] 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f689374cfd1] 16: /lib64/libpthread.so.0(+0x81ca) [0x7f68921b61ca] 2024-02-27T01:35:05.447 INFO:scrape:1 jobs: ['7574171'] 2024-02-27T01:35:05.447 INFO:scrape:suites: ['supported-all-distro/centos_8}', 'upgrade:reef-p2p/reef-p2p-parallel/{point-to-point-upgrade'] 2024-02-27T01:35:05.447 INFO:scrape: 2024-02-27T01:35:05.448 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 3 Fail ================================================================= [7574171] upgrade:reef-p2p/reef-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8} ----------------------------------------------------------------- time: 00:57:02 info: https://pulpito.ceph.com/yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7574171/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7574171/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ce47cf292e846a09196f7896c1e36e8 timeout expired in wait_until_healthy [7574174] upgrade:reef-p2p/reef-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} ----------------------------------------------------------------- time: 00:52:15 info: https://pulpito.ceph.com/yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7574174/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7574174/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b2cb68b94bd4712a491739fb849f520 Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json' Pass ================================================================= [7574172] upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} time: 03:25:18 info: https://pulpito.ceph.com/yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7574172/ [7574173] upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health} time: 02:58:24 info: https://pulpito.ceph.com/yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7574173/ [7574175] upgrade:reef-p2p/reef-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/reef 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health} time: 03:15:33 info: https://pulpito.ceph.com/yuriw-2024-02-26_21:53:51-upgrade:reef-p2p-wip-yuri8-testing-2024-02-22-0734-reef-distro-default-smithi/7574175/