2023-11-06T11:56:50.723 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-06T11:56:50.725 INFO:teuthology.results:subset: '111/120000'
2023-11-06T11:56:50.725 INFO:teuthology.results:seed: '8677'
2023-11-06T12:02:46.629 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-06T12:02:46.633 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-06T12:23:48.336 INFO:teuthology.results:Tests finished! gathering results...
2023-11-06T12:23:48.733 INFO:scrape:Found 9 jobs
2023-11-06T12:23:48.839 INFO:scrape:Found 1 distinct failure reasons
2023-11-06T12:23:48.840 INFO:scrape:Failure: Command failed on smithi083 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm'
2023-11-06T12:23:48.841 INFO:scrape:9 jobs: ['7448226', '7448218', '7448225', '7448223', '7448221', '7448220', '7448222', '7448219', '7448224']
2023-11-06T12:23:48.841 INFO:scrape:suites intersection: []
2023-11-06T12:23:48.842 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:reef-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'upgrade:reef-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0', 'upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04', 'workload/{ec-rados-default']
2023-11-06T12:23:48.842 INFO:scrape:
2023-11-06T12:23:48.843 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/
failed:  9
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7448218]  upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   00:14:34
info:   http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448218/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448218/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d8a9fe95e2724dd2a46f6df169964a19

    Command failed on smithi077 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7448219]  upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   00:08:38
info:   http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448219/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448219/
sentry: https://sentry.ceph.com/organizations/ceph/?query=57ecc7f988c1486d86381ea0dd0a3f15

    Command failed on smithi061 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7448220]  upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   00:12:22
info:   http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448220/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448220/
sentry: https://sentry.ceph.com/organizations/ceph/?query=29dfd62a26b2430cb7dacde46a61041c

    Command failed on smithi018 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7448221]  upgrade:reef-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   00:10:27
info:   http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448221/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448221/
sentry: https://sentry.ceph.com/organizations/ceph/?query=033156d67a26450494928bab30f809e4

    Command failed on smithi049 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7448222]  upgrade:reef-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   00:11:41
info:   http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448222/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448222/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5aa398b7ed004cf6909cc1a1a232b77d

    Command failed on smithi088 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7448223]  upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   00:15:15
info:   http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448223/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448223/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7a23d13cdf8e43949e1826ea7436cdfb

    Command failed on smithi082 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7448224]  upgrade:reef-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity}
-----------------------------------------------------------------
time:   00:14:59
info:   http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448224/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448224/
sentry: https://sentry.ceph.com/organizations/ceph/?query=94895a0957bf443fa7836bf8563e9076

    Command failed on smithi012 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7448225]  upgrade:reef-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
-----------------------------------------------------------------
time:   00:12:39
info:   http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448225/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448225/
sentry: https://sentry.ceph.com/organizations/ceph/?query=17a63bac5eb946a1839d52b3944c476f

    Command failed on smithi052 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'

[7448226]  upgrade:reef-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic}
-----------------------------------------------------------------
time:   00:08:12
info:   http://pulpito.ceph.com/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448226/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-05_15:11:55-upgrade:reef-x-main-distro-default-smithi/7448226/
sentry: https://sentry.ceph.com/organizations/ceph/?query=96ea6f1cd9af4bf596e6b83cceadad53

    Command failed on smithi083 with status 1: 'test -s
    /home/ubuntu/cephtest/cephadm && test $(stat -c%s
    /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x
    /home/ubuntu/cephtest/cephadm'