2023-05-07T12:28:27.211 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-07T12:28:27.212 INFO:teuthology.results:subset: '111/12000' 2023-05-07T12:28:27.212 INFO:teuthology.results:seed: '7127' 2023-05-07T12:37:42.716 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-07T12:37:42.733 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-08T00:38:39.237 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-05-08T00:38:39.817 INFO:scrape:Found 9 jobs 2023-05-08T00:38:45.175 INFO:scrape:Found 4 distinct failure reasons 2023-05-08T00:38:45.176 INFO:scrape:Dead: b'2023-05-08T00:38:39.535 INFO:journalctl@ceph.mon.b.smithi136.stdout:May 08 00:38:39 smithi136 bash[58606]: audit 1683506318.6484997 mon.a (mon.0) 17348 : audit 0 from=\'client.? 172.21.15.61:0/3465805032\' entity=\'client.admin\' cmd=[{"prefix": "mon versions"}]: dispatch' 2023-05-08T00:38:45.177 INFO:scrape:2 jobs: ['7264459', '7264454'] 2023-05-08T00:38:45.177 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2023-05-08T00:38:45.177 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0', 'upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04'] 2023-05-08T00:38:45.178 INFO:scrape: 2023-05-08T00:38:45.178 INFO:scrape:Failure: Command failed on smithi050 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8bde1772-ecd5-11ed-9b01-001a4aab830c -e sha1=1fd31bd5efc0285e085e8f305f8e4a7a9f22f84c -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2023-05-08T00:38:45.178 INFO:scrape:2 jobs: ['7264449', '7264457'] 2023-05-08T00:38:45.179 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'workload/{ec-rados-default'] 2023-05-08T00:38:45.179 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'workload/{ec-rados-default'] 2023-05-08T00:38:45.179 INFO:scrape: 2023-05-08T00:38:45.180 INFO:scrape:Dead: b'2023-05-08T00:36:54.192 DEBUG:teuthology.exit:Finished running handlers' 2023-05-08T00:38:45.180 INFO:scrape:4 jobs: ['7264446', '7264444', '7264452', '7264448'] 2023-05-08T00:38:45.180 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2023-05-08T00:38:45.180 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04'] 2023-05-08T00:38:45.181 INFO:scrape: 2023-05-08T00:38:45.181 INFO:scrape:Dead: b'2023-05-08T00:38:43.308 INFO:journalctl@ceph.mgr.y.smithi016.stdout:May 08 00:38:42 smithi016 ceph-eb539eac-ecd5-11ed-9b01-001a4aab830c-mgr-y[136907]: ::ffff:172.21.15.172 - - [08/May/2023:00:38:42] "GET /metrics HTTP/1.1" 200 245226 "" "Prometheus/2.33.4"' 2023-05-08T00:38:45.181 INFO:scrape:1 jobs: ['7264455'] 2023-05-08T00:38:45.182 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd_api', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8'] 2023-05-08T00:38:45.182 INFO:scrape: 2023-05-08T00:38:45.183 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/ failed: 2 dead: 1 running: 6 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7264449] upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:23:49 info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264449/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264449/ sentry: https://sentry.ceph.com/organizations/ceph/?query=835bf1e8b37a43648e5ea9f0bb9b1bb8 Command failed on smithi050 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8bde1772-ecd5-11ed-9b01-001a4aab830c -e sha1=1fd31bd5efc0285e085e8f305f8e4a7a9f22f84c -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' [7264457] upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:25:43 info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264457/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264457/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0bf5648cee1d41daa5d5580cba8ffd71 Command failed on smithi003 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2df85e4-ecd5-11ed-9b01-001a4aab830c -e sha1=1fd31bd5efc0285e085e8f305f8e4a7a9f22f84c -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' Dead ================================================================= [7264444] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264444/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264444/ hit max job timeout Running ================================================================= [7264446] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264446/ [7264448] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264448/ [7264452] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264452/ [7264454] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264454/ [7264455] upgrade:pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264455/ [7264459] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} info: http://pulpito.ceph.com/yuriw-2023-05-06_14:47:04-upgrade:pacific-x-reef-distro-default-smithi/7264459/