2021-12-19T18:29:15.965 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-19T18:29:15.967 INFO:teuthology.results:subset: '1/151000' 2021-12-19T18:29:15.967 INFO:teuthology.results:seed: '5063' 2021-12-19T18:35:39.609 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-19T18:35:39.615 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-12-20T06:36:31.727 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2021-12-20T06:36:33.472 INFO:scrape:Found 17 jobs 2021-12-20T06:36:35.046 INFO:scrape:Found 3 distinct failure reasons 2021-12-20T06:36:35.047 INFO:scrape:Dead: b'2021-12-20T06:36:32.975 INFO:teuthology.orchestra.run.smithi016.stdout: 5 cleanup 42157 sec' 2021-12-20T06:36:35.048 INFO:scrape:2 jobs: ['6572650', '6572644'] 2021-12-20T06:36:35.049 INFO:scrape:suites intersection: ['1-bootstrap/octopus', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2021-12-20T06:36:35.049 INFO:scrape:suites union: ['1-bootstrap/octopus', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2021-12-20T06:36:35.049 INFO:scrape: 2021-12-20T06:36:35.050 INFO:scrape:Failure: timeout expired in wait_until_healthy 2021-12-20T06:36:35.050 INFO:scrape:1 jobs: ['6572638'] 2021-12-20T06:36:35.051 INFO:scrape:suites: ['mon_election/classic', 'rados/cephadm/workunits/{agent/on', 'task/test_orch_cli}'] 2021-12-20T06:36:35.051 INFO:scrape: 2021-12-20T06:36:35.052 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi084 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2f8289cc42d98c773f754113826bbb51244a032f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2021-12-20T06:36:35.052 INFO:scrape:2 jobs: ['6572648', '6572643'] 2021-12-20T06:36:35.053 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}'] 2021-12-20T06:36:35.053 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zlib}', 'random-objectstore$/{bluestore-hybrid}', 'tasks/e2e}'] 2021-12-20T06:36:35.054 INFO:scrape: 2021-12-20T06:36:35.055 INFO:teuthology.results:Sending results to lflores@redhat.com: Test Run: lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/ failed: 3 dead: 0 running: 2 waiting: 0 queued: 0 passed: 12 Fail ================================================================= [6572638] rados/cephadm/workunits/{agent/on mon_election/classic task/test_orch_cli} ----------------------------------------------------------------- time: 00:17:01 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572638/ log: http://qa-proxy.ceph.com/teuthology/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572638/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3bc2ecbb38847de87c801a06c2a3b06 timeout expired in wait_until_healthy [6572643] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-hybrid} tasks/e2e} ----------------------------------------------------------------- time: 00:17:15 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572643/ log: http://qa-proxy.ceph.com/teuthology/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572643/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5644f63380a64ac487772df0c81fc7a3 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2f8289cc42d98c773f754113826bbb51244a032f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [6572648] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e} ----------------------------------------------------------------- time: 00:16:01 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572648/ log: http://qa-proxy.ceph.com/teuthology/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572648/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8529971181d48d7885bf940c776c60e Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi084 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2f8289cc42d98c773f754113826bbb51244a032f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' Running ================================================================= [6572644] rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572644/ [6572650] rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572650/ Pass ================================================================= [6572634] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:26:50 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572634/ [6572635] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:43:47 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572635/ [6572636] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:20 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572636/ [6572637] rados/cephadm/smoke-roleless/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:15:03 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572637/ [6572639] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:45:48 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572639/ [6572640] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:28:55 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572640/ [6572641] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/rados_api_tests} time: 00:26:17 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572641/ [6572642] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/3-node k8s/1.21 net/host rook/master} time: 00:31:20 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572642/ [6572645] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} time: 00:43:43 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572645/ [6572646] rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:43 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572646/ [6572647] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:21:52 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572647/ [6572649] rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_nfs} time: 00:28:23 info: http://pulpito.ceph.com/lflores-2021-12-19_18:26:29-rados-wip-bluestore-zero-detection-distro-default-smithi/6572649/