2022-12-13T18:26:53.820 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-13T18:26:53.821 INFO:teuthology.results:subset: '111/120000' 2022-12-13T18:26:53.821 INFO:teuthology.results:seed: '8789' 2022-12-13T19:19:49.866 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-13T19:19:49.892 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-12-14T01:01:15.940 INFO:teuthology.results:Tests finished! gathering results... 2022-12-14T01:01:16.375 INFO:scrape:Found 5 jobs 2022-12-14T01:01:17.241 INFO:scrape:Found 3 distinct failure reasons 2022-12-14T01:01:17.241 INFO:scrape:Failure: Command failed on smithi142 with status 1: 'sudo fuser -v /var/lib/dpkg/lock-frontend' 2022-12-14T01:01:17.241 INFO:scrape:1 jobs: ['7114843'] 2022-12-14T01:01:17.241 INFO:scrape:suites: ['1-ceph-install/pacific', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack'] 2022-12-14T01:01:17.242 INFO:scrape: 2022-12-14T01:01:17.242 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_cmpomap.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 CLS_RBD_GTEST_FILTER=\'*:-TestClsRbd.snapshots_namespaces\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cls/test_cls_cmpomap.sh' 2022-12-14T01:01:17.242 INFO:scrape:1 jobs: ['7114841'] 2022-12-14T01:01:17.242 INFO:scrape:suites: ['supported-all-distro/ubuntu_latest}', 'upgrade:pacific-p2p/pacific-p2p-parallel/{point-to-point-upgrade'] 2022-12-14T01:01:17.242 INFO:scrape: 2022-12-14T01:01:17.242 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2022-12-14T01:01:17.242 INFO:scrape:1 jobs: ['7114849'] 2022-12-14T01:01:17.242 INFO:scrape:suites: ['1-ceph-install/pacific', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack'] 2022-12-14T01:01:17.242 INFO:scrape: 2022-12-14T01:01:17.243 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [7114841] upgrade:pacific-p2p/pacific-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} ----------------------------------------------------------------- time: 00:34:22 info: http://pulpito.ceph.com/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/7114841/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/7114841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93091a2ae5a8495e9781d0aa58834413 Command failed (workunit test cls/test_cls_cmpomap.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 CLS_RBD_GTEST_FILTER=\'*:-TestClsRbd.snapshots_namespaces\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cls/test_cls_cmpomap.sh' [7114843] upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 05:19:43 info: http://pulpito.ceph.com/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/7114843/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/7114843/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b724555bb338429a952649b3bcdb47e0 Command failed on smithi142 with status 1: 'sudo fuser -v /var/lib/dpkg/lock-frontend' [7114849] upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 01:10:40 info: http://pulpito.ceph.com/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/7114849/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/7114849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00a8f2b944164bbf9b65ac15c23a2b95 Command failed on smithi079 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' Pass ================================================================= [7114845] upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health} time: 04:40:08 info: http://pulpito.ceph.com/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/7114845/ [7114847] upgrade:pacific-p2p/pacific-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/pacific 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health} time: 05:36:38 info: http://pulpito.ceph.com/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/7114847/