2024-01-30T22:54:19.393 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-30T22:54:19.393 INFO:teuthology.results:subset: '111/120000'
2024-01-30T22:54:19.393 INFO:teuthology.results:seed: '9853'
2024-01-31T00:00:50.050 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-31T00:00:50.189 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-01-31T02:01:59.392 INFO:teuthology.results:Tests finished! gathering results...
2024-01-31T02:02:00.056 INFO:scrape:Found 7 jobs
2024-01-31T02:02:00.921 INFO:scrape:Found 2 distinct failure reasons
2024-01-31T02:02:00.921 INFO:scrape:Failure: Command failed on smithi172 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch nautilus https://github.com/cbodley/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0'
2024-01-31T02:02:00.921 INFO:scrape:6 jobs: ['7539353', '7539351', '7539350', '7539354', '7539355', '7539349']
2024-01-31T02:02:00.922 INFO:scrape:suites intersection: ['1-ceph-install/nautilus', 'rgw_ragweed_prepare', 'start}', 'ubuntu_18.04}']
2024-01-31T02:02:00.922 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-workload/{blogbench', '3-thrash/default', '3-upgrade-sequence/upgrade-all', '4-pacific', '4-workload/{radosbench', '5-final-workload/{blogbench', '5-finish-upgrade', '6-pacific', '7-msgr2', '8-final-workload/{rbd-python', 'ec-rados-default', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_cls', 'rbd_import_export', 'readwrite', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api', 'test_rbd_python}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack']
2024-01-31T02:02:00.922 INFO:scrape:
2024-01-31T02:02:00.922 INFO:scrape:Failure: Failed to reconnect to smithi145
2024-01-31T02:02:00.922 INFO:scrape:1 jobs: ['7539352']
2024-01-31T02:02:00.922 INFO:scrape:suites: ['1-ceph-install/nautilus', '1.1-pg-log-overrides/short_pg_log', '2-workload/{blogbench', '3-upgrade-sequence/upgrade-mon-osd-mds', '4-pacific', '5-final-workload/{blogbench', 'ec-rados-default', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd_cls', 'rbd_import_export', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'start}', 'test_rbd_api', 'test_rbd_python}', 'ubuntu_18.04}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack']
2024-01-31T02:02:00.922 INFO:scrape:
2024-01-31T02:02:00.923 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/
failed:  7
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7539349]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   01:36:44
info:   https://pulpito.ceph.com/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539349/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539349/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cf6e693183f84f46ab4c1eb528e68bdf

    Command failed on smithi113 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch
    nautilus https://github.com/cbodley/ceph.git
    /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0'

[7539350]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:36:08
info:   https://pulpito.ceph.com/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539350/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539350/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f081b1b1670b457bbaf524d05901d777

    Command failed on smithi158 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch
    nautilus https://github.com/cbodley/ceph.git
    /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0'

[7539351]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   01:40:18
info:   https://pulpito.ceph.com/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539351/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539351/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9532569c726947d38cc144ab4431ecc0

    Command failed on smithi173 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch
    nautilus https://github.com/cbodley/ceph.git
    /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0'

[7539352]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539352/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539352/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3beab616892443ccbb4e296d2952f581

    Failed to reconnect to smithi145

[7539353]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   01:43:47
info:   https://pulpito.ceph.com/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539353/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539353/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5d4ee9c46cea442f925b511d7178d50b

    Command failed on smithi172 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch
    nautilus https://github.com/cbodley/ceph.git
    /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0'

[7539354]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-pacific 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:37:24
info:   https://pulpito.ceph.com/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539354/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539354/
sentry: https://sentry.ceph.com/organizations/ceph/?query=074b5db5840d4638beca8fb239a13219

    Command failed on smithi164 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch
    nautilus https://github.com/cbodley/ceph.git
    /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0'

[7539355]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-pacific 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   01:50:53
info:   https://pulpito.ceph.com/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539355/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_22:33:27-upgrade:nautilus-x-pacific-release-distro-default-smithi/7539355/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e47f4783f3ab45b88f2439cfc84b0cc9

    Command failed on smithi178 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch
    nautilus https://github.com/cbodley/ceph.git
    /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0'