2023-05-28T01:35:46.325 INFO:root:teuthology version: 0.0.1.dev111+gfa97ddd 2023-05-28T01:35:46.326 INFO:teuthology.results:seed: '6010' 2023-05-28T01:36:36.776 INFO:root:teuthology version: 0.0.1.dev111+gfa97ddd 2023-05-28T01:36:36.780 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-28T06:18:55.040 INFO:teuthology.results:Tests finished! gathering results... 2023-05-28T06:18:55.661 INFO:scrape:Found 6 jobs 2023-05-28T06:18:55.664 INFO:scrape:Found 2 distinct failure reasons 2023-05-28T06:18:55.664 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 CLS_RBD_GTEST_FILTER=\'*:-TestClsRbd.snapshots_namespaces\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cls/test_cls_rbd.sh' 2023-05-28T06:18:55.665 INFO:scrape:4 jobs: ['7288421', '7288424', '7288425', '7288423'] 2023-05-28T06:18:55.665 INFO:scrape:suites intersection: [] 2023-05-28T06:18:55.665 INFO:scrape:suites union: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/bluestore-comp', 'objectstore/bluestore-stupid', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/centos_8}', 'supported-all-distro/ubuntu_latest', 'supported-all-distro/ubuntu_latest}', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack'] 2023-05-28T06:18:55.666 INFO:scrape: 2023-05-28T06:18:55.666 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2023-05-28T06:18:55.666 INFO:scrape:1 jobs: ['7288426'] 2023-05-28T06:18:55.666 INFO:scrape:suites: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack'] 2023-05-28T06:18:55.667 INFO:scrape: 2023-05-28T06:18:55.667 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7288421] upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8} ----------------------------------------------------------------- time: 01:30:19 info: http://pulpito.ceph.com/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288421/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288421/ sentry: https://sentry.ceph.com/organizations/ceph/?query=873bc9871e92448199fb6e75881a1f9b Command failed (workunit test cls/test_cls_rbd.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 CLS_RBD_GTEST_FILTER=\'*:-TestClsRbd.snapshots_namespaces\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cls/test_cls_rbd.sh' [7288423] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 02:20:34 info: http://pulpito.ceph.com/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288423/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288423/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b91a4320fe6d4b18a50cf5e556270b2a Command failed (workunit test cls/test_cls_rbd.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7288424] upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} ----------------------------------------------------------------- time: 01:12:52 info: http://pulpito.ceph.com/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288424/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288424/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ed52198ef8b4b89bee0aa9adeb089fc Command failed (workunit test cls/test_cls_rbd.sh) on smithi192 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 CLS_RBD_GTEST_FILTER=\'*:-TestClsRbd.snapshots_namespaces\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cls/test_cls_rbd.sh' [7288425] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 03:30:41 info: http://pulpito.ceph.com/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288425/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288425/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9bae34cbbac24277a70c9be7b0be09ac Command failed (workunit test cls/test_cls_rbd.sh) on smithi125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7288426] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 02:10:23 info: http://pulpito.ceph.com/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288426/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8688fd4251146488adb0e245c6e0fbf Command failed on smithi023 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' Pass ================================================================= [7288422] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} time: 04:30:52 info: http://pulpito.ceph.com/teuthology-2023-05-28_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7288422/