2023-11-19T23:17:21.828 INFO:root:teuthology version: 0.0.1.dev205+g2442239
2023-11-19T23:17:21.829 INFO:teuthology.results:subset: '111/120000'
2023-11-19T23:17:21.829 INFO:teuthology.results:seed: '9627'
2023-11-19T23:31:02.005 INFO:root:teuthology version: 0.0.1.dev205+g2442239
2023-11-19T23:31:02.009 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-19T23:50:03.369 INFO:teuthology.results:Tests finished! gathering results...
2023-11-19T23:50:03.756 INFO:scrape:Found 9 jobs
2023-11-19T23:50:03.883 INFO:scrape:Found 1 distinct failure reasons
2023-11-19T23:50:03.883 INFO:scrape:Failure: Command failed on smithi061 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm'
2023-11-19T23:50:03.883 INFO:scrape:9 jobs: ['7462458', '7462460', '7462459', '7462461', '7462454', '7462453', '7462455', '7462457', '7462456']
2023-11-19T23:50:03.884 INFO:scrape:suites intersection: []
2023-11-19T23:50:03.884 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:reef-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'upgrade:reef-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0', 'upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04', 'workload/{ec-rados-default']
2023-11-19T23:50:03.884 INFO:scrape:
2023-11-19T23:50:03.885 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/
failed:  9
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7462453]  upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   00:16:42
info:   https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462453/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462453/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e25a289fa7444f759750b456d3ccc0ce

    Command failed on smithi132 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7462454]  upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   00:07:58
info:   https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462454/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462454/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9302c4ab066749ddacb3ee0bbe79b2d4

    Command failed on smithi005 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7462455]  upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   00:11:13
info:   https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462455/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462455/
sentry: https://sentry.ceph.com/organizations/ceph/?query=898af99939f64c519d538e1810bc9805

    Command failed on smithi100 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7462456]  upgrade:reef-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   00:13:05
info:   https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462456/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462456/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca7f9e0c2f6a4af4b8bc6315fd06f263

    Command failed on smithi049 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7462457]  upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   00:11:08
info:   https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462457/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462457/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e30d312326440d7a5027669879fa99b

    Command failed on smithi073 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7462458]  upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   00:15:43
info:   https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462458/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462458/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6e143dc5e0234882a9719011f02f04cd

    Command failed on smithi061 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7462459]  upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   00:13:42
info:   https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462459/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462459/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8f981145e26e4192b6b4e082bc7aa814

    Command failed on smithi019 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7462460]  upgrade:reef-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   00:09:44
info:   https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462460/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462460/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c0100a9ef93d4e16a6035a4b70deb733

    Command failed on smithi133 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7462461]  upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   00:07:51
info:   https://pulpito.ceph.com/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462461/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-18_14:01:34-upgrade:reef-x-main-distro-default-smithi/7462461/
sentry: https://sentry.ceph.com/organizations/ceph/?query=107ac7474ac0460ea352017e9de5ce0d

    Command failed on smithi142 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'