2024-02-04T19:38:32.023 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-04T19:38:32.023 INFO:teuthology.results:subset: '400/20000' 2024-02-04T19:38:32.023 INFO:teuthology.results:seed: '5841' 2024-02-04T19:54:54.758 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-04T19:54:54.775 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-04T21:04:00.258 INFO:teuthology.results:Tests finished! gathering results... 2024-02-04T21:04:01.105 INFO:scrape:Found 20 jobs 2024-02-04T21:04:01.665 INFO:scrape:Found 4 distinct failure reasons 2024-02-04T21:04:01.666 INFO:scrape:Failure: Command failed on smithi033 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9377a0f2510aab4987c9e75cbb31b132e383da56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c1e8b58-c399-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_4' 2024-02-04T21:04:01.666 INFO:scrape:2 jobs: ['7540280', '7540272'] 2024-02-04T21:04:01.666 INFO:scrape:suites intersection: ['agent/on'] 2024-02-04T21:04:01.666 INFO:scrape:suites union: ['0-nvme-loop', 'agent/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'start}', 'task/test_iscsi_container/{centos_8.stream_container_tools', 'test_iscsi_container}}'] 2024-02-04T21:04:01.666 INFO:scrape: 2024-02-04T21:04:01.666 INFO:scrape:Failure: Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9377a0f2510aab4987c9e75cbb31b132e383da56 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' 2024-02-04T21:04:01.666 INFO:scrape:1 jobs: ['7540269'] 2024-02-04T21:04:01.666 INFO:scrape:suites: ['mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{ubuntu_20.04}}'] 2024-02-04T21:04:01.667 INFO:scrape: 2024-02-04T21:04:01.667 INFO:scrape:Failure: Command failed (workunit test crush/crush-choose-args.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9377a0f2510aab4987c9e75cbb31b132e383da56 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/crush/crush-choose-args.sh' 2024-02-04T21:04:01.667 INFO:scrape:1 jobs: ['7540278'] 2024-02-04T21:04:01.667 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/crush}'] 2024-02-04T21:04:01.667 INFO:scrape: 2024-02-04T21:04:01.667 INFO:scrape:Failure: reached maximum tries (121) after waiting for 120 seconds 2024-02-04T21:04:01.667 INFO:scrape:1 jobs: ['7540264'] 2024-02-04T21:04:01.667 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun'] 2024-02-04T21:04:01.667 INFO:scrape: 2024-02-04T21:04:01.668 INFO:teuthology.results:Sending results to akupczyk@redhat.com: Test Run: akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 15 Fail ================================================================= [7540264] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:17:58 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540264/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540264/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1619c32e73db4f74b2f582e2741330a9 reached maximum tries (121) after waiting for 120 seconds [7540269] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} ----------------------------------------------------------------- time: 00:10:37 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540269/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540269/ sentry: https://sentry.ceph.com/organizations/ceph/?query=178d7cd4cacc419fa8ad033d7e16bf32 Command failed (workunit test cephtool/test_kvstore_tool.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9377a0f2510aab4987c9e75cbb31b132e383da56 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' [7540272] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:09:50 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540272/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540272/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b98efa6771d540379e37a63833ec9195 Command failed on smithi028 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:9377a0f2510aab4987c9e75cbb31b132e383da56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1dae6bea-c398-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi028:/dev/nvme4n1' [7540278] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} ----------------------------------------------------------------- time: 00:17:04 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540278/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540278/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0301b97fb65e4c31bd198fc1f13ed181 Command failed (workunit test crush/crush-choose-args.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9377a0f2510aab4987c9e75cbb31b132e383da56 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/crush/crush-choose- args.sh' [7540280] rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} ----------------------------------------------------------------- time: 00:12:04 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540280/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540280/ sentry: https://sentry.ceph.com/organizations/ceph/?query=861530a715f8434b80ff7ceb13a053af Command failed on smithi033 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:9377a0f2510aab4987c9e75cbb31b132e383da56 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c1e8b58-c399-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_4' Pass ================================================================= [7540261] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 01:13:13 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540261/ [7540262] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} time: 00:11:39 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540262/ [7540263] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/dedup-io-mixed} time: 00:13:06 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540263/ [7540265] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_8} tasks/failover} time: 00:14:56 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540265/ [7540266] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} time: 00:19:08 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540266/ [7540267] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_20.04} tasks/readwrite} time: 00:13:10 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540267/ [7540268] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:34:12 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540268/ [7540270] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:09:59 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540270/ [7540271] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:33:04 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540271/ [7540273] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:02 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540273/ [7540274] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/snaps-few-objects} time: 00:29:45 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540274/ [7540275] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:28:36 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540275/ [7540276] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:20:41 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540276/ [7540277] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:48 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540277/ [7540279] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:36:40 info: https://pulpito.ceph.com/akupczyk-2024-01-31_13:35:31-rados-wip-aclamk-bs-multi-label-distro-default-smithi/7540279/