2024-01-20T23:27:03.768 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-20T23:27:03.768 INFO:teuthology.results:subset: '111/120000' 2024-01-20T23:27:03.769 INFO:teuthology.results:seed: '5950' 2024-01-20T23:33:09.316 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-20T23:33:09.322 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-21T02:11:20.560 INFO:teuthology.results:Tests finished! gathering results... 2024-01-21T02:11:21.235 INFO:scrape:Found 10 jobs 2024-01-21T02:11:23.601 INFO:scrape:Found 3 distinct failure reasons 2024-01-21T02:11:23.602 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' 2024-01-21T02:11:23.602 INFO:scrape:2 jobs: ['7523280', '7523275'] 2024-01-21T02:11:23.602 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-01-21T02:11:23.602 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'upgrade:quincy-x/parallel/{0-random-distro$/{ubuntu_20.04}', 'workload/{ec-rados-default'] 2024-01-21T02:11:23.602 INFO:scrape: 2024-01-21T02:11:23.603 INFO:scrape:Failure: timeout expired in wait_until_healthy 2024-01-21T02:11:23.603 INFO:scrape:1 jobs: ['7523273'] 2024-01-21T02:11:23.603 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04'] 2024-01-21T02:11:23.603 INFO:scrape: 2024-01-21T02:11:23.603 INFO:scrape:Failure: Command failed on smithi161 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3141a96a-b7ef-11ee-95b0-87774f69a715 -- ceph mon dump -f json' 2024-01-21T02:11:23.603 INFO:scrape:1 jobs: ['7523277'] 2024-01-21T02:11:23.603 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd-import-export', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0'] 2024-01-21T02:11:23.603 INFO:scrape: 2024-01-21T02:11:23.604 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7523273] upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:51:30 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523273/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523273/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0317a94e5db6447096b9e03f4841fd9a timeout expired in wait_until_healthy [7523275] upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:29:17 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523275/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523275/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d6c3cb70a0e4f33b492083403aef637 Command failed (workunit test rbd/test_librbd_python.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' [7523277] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:19:36 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523277/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523277/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93c894db8944446e8c4a3aa04d697e08 Command failed on smithi161 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3141a96a-b7ef-11ee-95b0-87774f69a715 -- ceph mon dump -f json' [7523280] upgrade:quincy-x/parallel/{0-random-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:39:22 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523280/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523280/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e980d5179e674623bb7c5f213ae3a3dc Command failed (workunit test rbd/test_librbd_python.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' Pass ================================================================= [7523272] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:07:57 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523272/ [7523274] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:08:13 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523274/ [7523276] upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:04:19 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523276/ [7523278] upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:16:47 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523278/ [7523279] upgrade:quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_20.04} time: 00:08:47 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523279/ [7523281] upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:29:04 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:32:15-upgrade:quincy-x-main-distro-default-smithi/7523281/