2024-02-23T19:53:17.024 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-23T19:53:17.024 INFO:teuthology.results:subset: '111/120000'
2024-02-23T19:53:17.024 INFO:teuthology.results:seed: '273'
2024-02-23T19:53:22.117 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-02-23T19:53:22.117 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-02-24T07:54:16.261 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2024-02-24T07:54:16.900 INFO:scrape:Found 9 jobs
2024-02-24T07:54:17.039 INFO:scrape:Found 5 distinct failure reasons
2024-02-24T07:54:17.040 INFO:scrape:Dead: b'2024-02-24T07:54:16.805 INFO:journalctl@ceph.mgr.x.smithi138.stdout:Feb 24 07:54:16 smithi138 ceph-31692c92-d289-11ee-95c1-87774f69a715-mgr-x[69226]: 2024-02-24T07:54:16.520+0000 7f6112e8f700 -1 failed to decode message of type 45 v1: End of buffer [buffer:2]'
2024-02-24T07:54:17.040 INFO:scrape:2 jobs: ['7572125', '7572129']
2024-02-24T07:54:17.040 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc']
2024-02-24T07:54:17.040 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc']
2024-02-24T07:54:17.040 INFO:scrape:
2024-02-24T07:54:17.040 INFO:scrape:Dead: b'2024-02-24T07:54:15.196 INFO:journalctl@ceph.mgr.y.smithi133.stdout:Feb 24 07:54:14 smithi133 bash[21591]: ::ffff:172.21.15.142 - - [24/Feb/2024:07:54:14] "GET /metrics HTTP/1.1" 200 33931 "" "Prometheus/2.43.0"'
2024-02-24T07:54:17.040 INFO:scrape:1 jobs: ['7572128']
2024-02-24T07:54:17.040 INFO:scrape:suites: ['0-start', '1-tasks', 'mon_election/classic', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:reef-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'workload/{ec-rados-default']
2024-02-24T07:54:17.040 INFO:scrape:
2024-02-24T07:54:17.041 INFO:scrape:Dead: b'2024-02-24T07:54:16.975 INFO:journalctl@ceph.mon.a.smithi088.stdout:Feb 24 07:54:16 smithi088 ceph-mon[29076]: from=\'mgr.25072 172.21.15.132:0/957150015\' entity=\'mgr.x\' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch'
2024-02-24T07:54:17.041 INFO:scrape:4 jobs: ['7572135', '7572124', '7572127', '7572131']
2024-02-24T07:54:17.041 INFO:scrape:suites intersection: []
2024-02-24T07:54:17.041 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:reef-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade:reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc', 'workload/{ec-rados-default']
2024-02-24T07:54:17.041 INFO:scrape:
2024-02-24T07:54:17.041 INFO:scrape:Dead: b'2024-02-24T07:54:07.250 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Feb 24 07:54:06 smithi112 bash[21597]: ::ffff:172.21.15.175 - - [24/Feb/2024:07:54:06] "GET /metrics HTTP/1.1" 200 33779 "" "Prometheus/2.43.0"'
2024-02-24T07:54:17.041 INFO:scrape:1 jobs: ['7572133']
2024-02-24T07:54:17.041 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd-cls', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04']
2024-02-24T07:54:17.041 INFO:scrape:
2024-02-24T07:54:17.041 INFO:scrape:Dead: b'2024-02-24T07:54:13.046 INFO:journalctl@ceph.mgr.x.smithi152.stdout:Feb 24 07:54:12 smithi152 bash[52466]: debug 2024-02-24T07:54:12.967+0000 7fbe9a157700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-24T06:54:12.969104+0000)'
2024-02-24T07:54:17.041 INFO:scrape:1 jobs: ['7572126']
2024-02-24T07:54:17.042 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd_api', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04']
2024-02-24T07:54:17.042 INFO:scrape:
2024-02-24T07:54:17.042 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/
failed:  0
dead:    0
running: 9
waiting: 0
queued:  0
passed:  0


Running
=================================================================
[7572124] upgrade:reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
info:   https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/7572124/

[7572125] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
info:   https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/7572125/

[7572126] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
info:   https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/7572126/

[7572127] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
info:   https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/7572127/

[7572128] upgrade:reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
info:   https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/7572128/

[7572129] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
info:   https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/7572129/

[7572131] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
info:   https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/7572131/

[7572133] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic}
info:   https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/7572133/

[7572135] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity}
info:   https://pulpito.ceph.com/yuriw-2024-02-23_19:52:28-upgrade:reef-x-main-distro-default-smithi/7572135/