2023-05-02T15:20:43.394 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-02T15:20:43.395 INFO:teuthology.results:subset: '111/120000' 2023-05-02T15:20:43.395 INFO:teuthology.results:seed: '5280' 2023-05-02T16:30:54.705 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-02T16:30:54.739 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-02T18:43:04.810 INFO:teuthology.results:Tests finished! gathering results... 2023-05-02T18:43:05.588 INFO:scrape:Found 9 jobs 2023-05-02T18:43:06.620 INFO:scrape:Found 3 distinct failure reasons 2023-05-02T18:43:06.621 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_refcount.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh' 2023-05-02T18:43:06.621 INFO:scrape:2 jobs: ['7256754', '7256762'] 2023-05-02T18:43:06.622 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2023-05-02T18:43:06.622 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.4_container_tools_3.0}', 'workload/{ec-rados-default'] 2023-05-02T18:43:06.622 INFO:scrape: 2023-05-02T18:43:06.623 INFO:scrape:Failure: timeout expired in wait_until_healthy 2023-05-02T18:43:06.623 INFO:scrape:1 jobs: ['7256756'] 2023-05-02T18:43:06.623 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/rbd-cls', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun'] 2023-05-02T18:43:06.623 INFO:scrape: 2023-05-02T18:43:06.624 INFO:scrape:Failure: Command failed on smithi136 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 7e7d940a-e900-11ed-9b01-001a4aab830c -- ceph osd dump --format=json' 2023-05-02T18:43:06.624 INFO:scrape:1 jobs: ['7256748'] 2023-05-02T18:43:06.624 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8'] 2023-05-02T18:43:06.625 INFO:scrape: 2023-05-02T18:43:06.626 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7256748] upgrade:pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:24:09 info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256748/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a90aaef1893547aab62c433d71daf2d7 Command failed on smithi136 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell --fsid 7e7d940a-e900-11ed-9b01-001a4aab830c -- ceph osd dump --format=json' [7256754] upgrade:pacific-x/parallel/{0-random-distro$/{rhel_8.4_container_tools_3.0} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:44:47 info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256754/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256754/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d30ad2c5a2c74d228892f8aef34d952b Command failed (workunit test cls/test_cls_refcount.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh' [7256756] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:48:14 info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256756/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18c3568fec364b88b0c0ce295a39fd86 timeout expired in wait_until_healthy [7256762] upgrade:pacific-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:39:00 info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256762/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256762/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f90fb58c0ed4b30b846721c207e7f30 Command failed (workunit test cls/test_cls_refcount.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh' Pass ================================================================= [7256750] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:21:20 info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256750/ [7256752] upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:14:28 info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256752/ [7256758] upgrade:pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 01:53:20 info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256758/ [7256760] upgrade:pacific-x/stress-split/{0-distro/rhel_8.4_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} time: 02:18:46 info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256760/ [7256764] upgrade:pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} time: 02:09:30 info: http://pulpito.ceph.com/teuthology-2023-04-28_14:23:02-upgrade:pacific-x-quincy-distro-default-smithi/7256764/