2023-12-06T18:47:47.963 INFO:root:teuthology version: 0.0.1.dev214+gf2de108
2023-12-06T18:47:47.963 INFO:teuthology.results:subset: '111/120000'
2023-12-06T18:47:47.964 INFO:teuthology.results:seed: '8126'
2023-12-06T18:56:17.954 INFO:root:teuthology version: 0.0.1.dev214+gf2de108
2023-12-06T18:56:17.958 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-12-07T06:57:14.700 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-12-07T06:57:15.372 INFO:scrape:Found 9 jobs
2023-12-07T06:57:15.468 INFO:scrape:Found 4 distinct failure reasons
2023-12-07T06:57:15.468 INFO:scrape:Dead: b'2023-12-07T06:57:13.440 INFO:journalctl@ceph.mgr.y.smithi060.stdout:Dec 07 06:57:13 smithi060 bash[28504]: ::ffff:172.21.15.145 - - [07/Dec/2023:06:57:13] "GET /metrics HTTP/1.1" 200 33848 "" "Prometheus/2.43.0"'
2023-12-07T06:57:15.468 INFO:scrape:1 jobs: ['7480674']
2023-12-07T06:57:15.468 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04']
2023-12-07T06:57:15.468 INFO:scrape:
2023-12-07T06:57:15.469 INFO:scrape:Dead: b'2023-12-07T06:57:14.777 INFO:journalctl@ceph.mon.a.smithi174.stdout:Dec 07 06:57:14 smithi174 ceph-mon[111990]: from=\'mgr.14937 172.21.15.195:0/1726303364\' entity=\'mgr.x\' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch'
2023-12-07T06:57:15.469 INFO:scrape:6 jobs: ['7480668', '7480670', '7480673', '7480672', '7480667', '7480669']
2023-12-07T06:57:15.469 INFO:scrape:suites intersection: []
2023-12-07T06:57:15.469 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd_api', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:reef-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04', 'workload/{ec-rados-default']
2023-12-07T06:57:15.469 INFO:scrape:
2023-12-07T06:57:15.469 INFO:scrape:Dead: b'2023-12-07T06:54:40.492 DEBUG:teuthology.exit:Finished running handlers'
2023-12-07T06:57:15.469 INFO:scrape:1 jobs: ['7480666']
2023-12-07T06:57:15.469 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8']
2023-12-07T06:57:15.469 INFO:scrape:
2023-12-07T06:57:15.470 INFO:scrape:Dead: b'2023-12-07T06:57:15.435 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Dec 07 06:57:15 smithi186 conmon[159837]: 2023-12-07T06:57:15.348+0000 7f786e918700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-07T05:57:15.348421+0000)'
2023-12-07T06:57:15.470 INFO:scrape:1 jobs: ['7480671']
2023-12-07T06:57:15.470 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd-import-export', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0']
2023-12-07T06:57:15.470 INFO:scrape:
2023-12-07T06:57:15.471 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/
failed:  0
dead:    0
running: 9
waiting: 0
queued:  0
passed:  0


Running
=================================================================
[7480666] upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
info:   https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/7480666/

[7480667] upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
info:   https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/7480667/

[7480668] upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
info:   https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/7480668/

[7480669] upgrade:reef-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
info:   https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/7480669/

[7480670] upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
info:   https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/7480670/

[7480671] upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
info:   https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/7480671/

[7480672] upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
info:   https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/7480672/

[7480673] upgrade:reef-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
info:   https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/7480673/

[7480674] upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
info:   https://pulpito.ceph.com/yuriw-2023-12-06_18:45:46-upgrade:reef-x-wip-yuri10-testing-2023-12-05-1105-distro-default-smithi/7480674/