2023-07-22T12:15:10.820 INFO:root:teuthology version: 0.0.1.dev158+g1426592 2023-07-22T12:15:10.822 INFO:teuthology.results:subset: '111/120000' 2023-07-22T12:15:10.822 INFO:teuthology.results:seed: '8766' 2023-07-22T13:08:35.713 INFO:root:teuthology version: 0.0.1.dev158+g1426592 2023-07-22T13:08:35.737 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-23T00:59:40.864 INFO:teuthology.results:Tests finished! gathering results... 2023-07-23T00:59:46.106 INFO:scrape:Found 27 jobs 2023-07-23T00:59:50.366 INFO:scrape:Found 4 distinct failure reasons 2023-07-23T00:59:50.367 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-07-23T00:59:50.367 INFO:scrape:13 jobs: ['7346700', '7346669', '7346695', '7346694', '7346691', '7346699', '7346682', '7346677', '7346696', '7346693', '7346679', '7346687', '7346692'] 2023-07-23T00:59:50.367 INFO:scrape:suites intersection: [] 2023-07-23T00:59:50.368 INFO:scrape:suites union: ['0-roles', '0-start', '1-ceph-install/octopus', '1-start', '1-tasks', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-partial-upgrade/firsthalf', '3-stress-tasks/{radosbench', '3-thrash/default', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', '4-workload/{radosbench', '5-finish-upgrade', '6-quincy', '8-final-workload/{rbd-python', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api}}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools', 'upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8', 'upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04', 'upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0', 'workload/{ec-rados-default'] 2023-07-23T00:59:50.368 INFO:scrape: 2023-07-23T00:59:50.368 INFO:scrape:Dead: b'2023-07-23T00:35:11.971 DEBUG:teuthology.exit:Finished running handlers' 2023-07-23T00:59:50.369 INFO:scrape:1 jobs: ['7346684'] 2023-07-23T00:59:50.369 INFO:scrape:suites: ['1-octopus-install/octopus', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-hybrid', '4-ec-workload/{rados-ec-workload', '5-finish-upgrade', '6-quincy', '7-final-workload', 'mon_election/classic', 'rbd-ec-workload}', 'start}', 'thrashosds-health', 'ubuntu_20.04}', 'upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack'] 2023-07-23T00:59:50.369 INFO:scrape: 2023-07-23T00:59:50.370 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=octopus 2023-07-23T00:59:50.370 INFO:scrape:2 jobs: ['7346697', '7346683'] 2023-07-23T00:59:50.370 INFO:scrape:suites intersection: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-07-23T00:59:50.370 INFO:scrape:suites union: ['frontend', 'overrides', 'realm', 'tasks', 'upgrade/primary}', 'upgrade/secondary}', 'upgrade:octopus-x/rgw-multisite/{clusters'] 2023-07-23T00:59:50.371 INFO:scrape: 2023-07-23T00:59:50.371 INFO:scrape:Failure: Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0761a874-288f-11ee-9b34-001a4aab830c -e sha1=6e26ecceb84bd481a9301eb10c077bd79a8a0852 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-07-23T00:59:50.371 INFO:scrape:2 jobs: ['7346688', '7346689'] 2023-07-23T00:59:50.372 INFO:scrape:suites intersection: [] 2023-07-23T00:59:50.372 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/classic', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api}}', 'upgrade-sequence', 'upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0', 'upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools', 'workload/{ec-rados-default'] 2023-07-23T00:59:50.372 INFO:scrape: 2023-07-23T00:59:50.375 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/ failed: 17 dead: 1 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [7346669] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:44:53 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346669/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346669/ sentry: https://sentry.ceph.com/organizations/ceph/?query=add94ef8a5bd41b6a615c5aca1676b78 Command failed (workunit test cls/test_cls_rbd.sh) on smithi130 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346677] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:37:04 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346677/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346677/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd252ee0c4434489b70068905dc4ec69 Command failed (workunit test cls/test_cls_rbd.sh) on smithi053 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346679] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:12 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346679/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346679/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b57020d5dfe6462bb995487f2f621e24 Command failed (workunit test cls/test_cls_rbd.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346682] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:33 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346682/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346682/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a4dcf9a5e714e6297059019cc04bf6e Command failed (workunit test cls/test_cls_rbd.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346683] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/primary} ----------------------------------------------------------------- time: 00:07:51 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346683/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346683/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a28083b897214e5583ffa36874b1b7cc Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7346687] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:34:05 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346687/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346687/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fab90c0d0f8445978fdfd207bc021f1a Command failed (workunit test cls/test_cls_rbd.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346688] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_3.0 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 01:03:35 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346688/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346688/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1523a3b8bab644dba358f8d4c60730ee Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0761a874-288f-11ee-9b34-001a4aab830c -e sha1=6e26ecceb84bd481a9301eb10c077bd79a8a0852 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7346689] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:53:49 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346689/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346689/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d811e1553db4394aedf553303b766d1 Command failed on smithi003 with status 95: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e96ebc8-288e-11ee-9b34-001a4aab830c -e sha1=6e26ecceb84bd481a9301eb10c077bd79a8a0852 -- bash -c 'ceph orch ps'" [7346691] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 01:35:35 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346691/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346691/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f1148afd0f842b59c18216ebf01b92e Command failed (workunit test cls/test_cls_rbd.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346692] upgrade:octopus-x/parallel/{0-distro/rhel_8.4_container_tools_rhel8 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:37:43 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346692/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346692/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31f58992e6f244878b5cef177f0af77f Command failed (workunit test cls/test_cls_rbd.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346693] upgrade:octopus-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:55:23 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346693/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346693/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fdfa491a949f46998194c3eb1532add9 Command failed (workunit test cls/test_cls_rbd.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346694] upgrade:octopus-x/parallel/{0-distro/ubuntu_20.04 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:36:55 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346694/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346694/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2190336f0bfb4f0f88f31c0084ddcbe5 Command failed (workunit test cls/test_cls_rbd.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346695] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:30:44 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346695/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346695/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aab9821e617d459c852b29c3d6c0537d Command failed (workunit test cls/test_cls_rbd.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346696] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:40:12 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346696/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346696/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5fb83676963459c9e4965df7dba71db Command failed (workunit test cls/test_cls_rbd.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346697] upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} ----------------------------------------------------------------- time: 00:07:47 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346697/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346697/ sentry: https://sentry.ceph.com/organizations/ceph/?query=826c4079a2a544ff930c096129fce936 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref= octopus [7346699] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 03:14:44 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346699/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346699/ sentry: https://sentry.ceph.com/organizations/ceph/?query=815893e6751b40f8a852a15830e6c27d Command failed (workunit test cls/test_cls_rbd.sh) on smithi191 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7346700] upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools_crun 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} ----------------------------------------------------------------- time: 00:41:17 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346700/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346700/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65e2b2cfc62d4dd09455f0339a16af12 Command failed (workunit test cls/test_cls_rbd.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' Dead ================================================================= [7346684] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346684/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346684/ hit max job timeout Pass ================================================================= [7346671] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-hybrid 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 03:00:21 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346671/ [7346674] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_20.04} time: 02:24:59 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346674/ [7346675] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:53:32 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346675/ [7346681] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:19:34 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346681/ [7346685] upgrade:octopus-x/stress-split-no-cephadm/{0-cluster/{openstack start} 1-ceph-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-quincy 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-hybrid thrashosds-health ubuntu_20.04} time: 04:09:52 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346685/ [7346686] upgrade:octopus-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:59:33 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346686/ [7346690] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/connectivity thrashosds-health ubuntu_20.04} time: 02:19:16 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346690/ [7346698] upgrade:octopus-x/stress-split-erasure-code-no-cephadm/{0-cluster/{openstack start} 1-octopus-install/octopus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-quincy 7-final-workload mon_election/classic thrashosds-health ubuntu_20.04} time: 01:44:26 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346698/ [7346701] upgrade:octopus-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:08:24 info: http://pulpito.ceph.com/teuthology-2023-07-21_14:22:11-upgrade:octopus-x-quincy-distro-default-smithi/7346701/