2021-02-07T09:39:52.678 INFO:root:teuthology version: 1.1.0-17c09965
2021-02-07T09:39:52.678 INFO:teuthology.results:subset: '1/5000'
2021-02-07T09:39:52.678 INFO:teuthology.results:seed: '3263'
2021-02-13T08:46:12.760 INFO:root:teuthology version: 1.1.0-64d5f180
2021-02-13T08:46:12.807 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-02-13T20:46:31.610 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2021-02-13T20:46:50.473 INFO:scrape:Found 194 jobs
2021-02-13T20:47:09.519 INFO:scrape:Found 14 distinct failure reasons
2021-02-13T20:47:09.520 INFO:scrape:Dead: b'2021-02-07T22:21:55.211 DEBUG:teuthology.exit:Finished running handlers'
2021-02-13T20:47:09.520 INFO:scrape:114 jobs
2021-02-13T20:47:09.521 INFO:scrape:suites intersection: []
2021-02-13T20:47:09.521 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/3-mons', 'clusters/9-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/mon-delay', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'orchestrator_cli', 'rados', 'rados/basic/{ceph', 'rados/cephadm/orchestrator_cli/{2-node-mgr', 'rados/dashboard/{clusters/{2-node-mgr}', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/monthrash/{ceph', 'rados/objectstore/{backends/filestore-idempotent', 'rados/objectstore/{backends/filestore-idempotent-aio-journal', 'rados/singleton-bluestore/{all/cephtool', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/singleton-nomsgr/{all/balancer', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'rados/singleton-nomsgr/{all/export-after-evict', 'rados/singleton-nomsgr/{all/full-tiering', 'rados/singleton-nomsgr/{all/health-warnings', 'rados/singleton-nomsgr/{all/large-omap-object-warnings', 'rados/singleton-nomsgr/{all/librados_hello_world', 'rados/singleton-nomsgr/{all/multi-backfill-reject', 'rados/singleton-nomsgr/{all/osd_stale_reads', 'rados/singleton-nomsgr/{all/version-number-sanity', 'rados/singleton/{all/admin-socket', 'rados/singleton/{all/divergent_priors', 'rados/singleton/{all/dump-stuck', 'rados/singleton/{all/lost-unfound', 'rados/singleton/{all/lost-unfound-delete', 'rados/singleton/{all/max-pg-per-osd.from-primary', 'rados/singleton/{all/max-pg-per-osd.from-replica', 'rados/singleton/{all/mon-config', 'rados/singleton/{all/mon-config-keys', 'rados/singleton/{all/osd-backfill', 'rados/singleton/{all/osd-recovery-incomplete', 'rados/singleton/{all/pg-removal-interruption', 'rados/singleton/{all/rebuild-mondb', 'rados/singleton/{all/test-crash', 'rados/singleton/{all/test_envlibrados_for_rocksdb', 'rados/singleton/{all/thrash-eio', 'rados/singleton/{all/thrash-rados/{thrash-rados', 'rados/singleton/{all/watch-notify-same-primary', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'tasks/crash}', 'tasks/dashboard}', 'tasks/failover}', 'tasks/mon_recovery', 'tasks/prometheus}', 'tasks/rados_api_tests', 'tasks/rados_api_tests}', 'tasks/rados_cls_all', 'tasks/rados_cls_all}', 'tasks/rados_stress_watch}', 'tasks/rados_striper}', 'tasks/rados_workunit_loadgen_big}', 'tasks/rados_workunit_loadgen_mostlyread}', 'thrashers/careful', 'thrashers/default', 'thrashers/force-sync-many', 'thrashers/many', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/one', 'thrashers/pggrow', 'thrashers/sync', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'workloads/admin_socket_objecter_requests}', 'workloads/pool-create-delete}', 'workloads/rados_api_tests}', 'workloads/rados_mon_osdmap_prune}', 'workloads/rados_mon_workunits}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/redirect_promote_tests}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/write_fadvise_dontneed}']
2021-02-13T20:47:09.522 INFO:scrape:
2021-02-13T20:47:09.522 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json
2021-02-13T20:47:09.522 INFO:scrape:3 jobs: ['5851030', '5850913', '5850973']
2021-02-13T20:47:09.522 INFO:scrape:suites intersection: ['msgr/async-v1only', 'no_pools', 'rados', 'tasks/mon_clock_no_skews}']
2021-02-13T20:47:09.522 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async-v1only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/multimon/{clusters/21', 'rados/multimon/{clusters/6', 'rados/multimon/{clusters/9', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/mon_clock_no_skews}']
2021-02-13T20:47:09.522 INFO:scrape:
2021-02-13T20:47:09.522 INFO:scrape:Failure: timeout expired in wait_until_healthy
2021-02-13T20:47:09.523 INFO:scrape:22 jobs: ['5850946', '5850982', '5850882', '5850953', '5851014', '5850929', '5850996', '5851031', '5851040', '5850998', '5850934', '5851043', '5850958', '5850893', '5851046', '5850900', '5850948', '5851038', '5850897', '5851004', '5850895', '5851056']
2021-02-13T20:47:09.523 INFO:scrape:suites intersection: ['fixed-2']
2021-02-13T20:47:09.523 INFO:scrape:suites union: ['fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v2only', 'rados/cephadm/smoke/{distro/centos_8.0', 'rados/cephadm/smoke/{distro/centos_latest', 'rados/cephadm/smoke/{distro/rhel_8.0', 'rados/cephadm/smoke/{distro/rhel_latest', 'rados/cephadm/with-work/{distro/centos_8.0', 'rados/cephadm/with-work/{distro/centos_latest', 'rados/cephadm/with-work/{distro/rhel_8.0', 'rados/cephadm/with-work/{distro/rhel_latest', 'start', 'start}', 'tasks/rados_api_tests}', 'tasks/rados_python}']
2021-02-13T20:47:09.523 INFO:scrape:
2021-02-13T20:47:09.523 INFO:scrape:Failure: Command crashed: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap'
2021-02-13T20:47:09.523 INFO:scrape:15 jobs: ['5850871', '5851057', '5851016', '5850925', '5850963', '5851048', '5850885', '5850941', '5850969', '5851033', '5850956', '5850896', '5851001', '5851023', '5851060']
2021-02-13T20:47:09.523 INFO:scrape:suites intersection: []
2021-02-13T20:47:09.524 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/dashboard/{clusters/{2-node-mgr}', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/objectstore/{backends/alloc-hint', 'rados/objectstore/{backends/ceph_objectstore_tool', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'rados/singleton/{all/resolve_stuck_peering', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'tasks/dashboard}', 'tasks/module_selftest}', 'tasks/rados_workunit_loadgen_mix}', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}', 'workloads/radosbench}', 'workloads/redirect_promote_tests}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/write_fadvise_dontneed}']
2021-02-13T20:47:09.524 INFO:scrape:
2021-02-13T20:47:09.524 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi086 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c9fbc794811a1de66ee254a1f9078372839e4f3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'
2021-02-13T20:47:09.524 INFO:scrape:2 jobs: ['5850979', '5850875']
2021-02-13T20:47:09.524 INFO:scrape:suites intersection: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}']
2021-02-13T20:47:09.524 INFO:scrape:suites union: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}']
2021-02-13T20:47:09.524 INFO:scrape:
2021-02-13T20:47:09.524 INFO:scrape:Failure: Command failed (workunit test misc/network-ping.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c9fbc794811a1de66ee254a1f9078372839e4f3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh'
2021-02-13T20:47:09.525 INFO:scrape:1 jobs: ['5850915']
2021-02-13T20:47:09.525 INFO:scrape:suites: ['rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{centos_8}', 'workloads/misc}']
2021-02-13T20:47:09.525 INFO:scrape:
2021-02-13T20:47:09.525 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds
2021-02-13T20:47:09.525 INFO:scrape:6 jobs: ['5851007', '5850989', '5850887', '5850930', '5850908', '5851054']
2021-02-13T20:47:09.525 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start']
2021-02-13T20:47:09.525 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_8.0}', 'distro$/{centos_latest}', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start']
2021-02-13T20:47:09.525 INFO:scrape:
2021-02-13T20:47:09.526 INFO:scrape:Dead: b'2021-02-13T20:46:41.497 INFO:teuthology.orchestra.run.smithi049.stderr:INFO  2021-02-13 20:46:41,496 [shard 0] ms - [client.?(temp_mon_client) - >> mon.? v2:172.21.15.49:3300/0] write_event: delay ...'
2021-02-13T20:47:09.526 INFO:scrape:2 jobs: ['5851061', '5851063']
2021-02-13T20:47:09.526 INFO:scrape:suites intersection: ['rados']
2021-02-13T20:47:09.526 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/version-number-sanity', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/radosbench-high-concurrency}']
2021-02-13T20:47:09.526 INFO:scrape:
2021-02-13T20:47:09.526 INFO:scrape:Failure: Command failed on smithi014 with status 1: "sudo cephadm --image quay.ceph.io/ceph-ci/ceph:09bdf42b0f05c3488246e70bd45edf8a11db8389-crimson -v bootstrap --fsid a4a71a78-692a-11eb-8fdc-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.14:6789]' && sudo chmod +r /etc/ceph/ceph.client.admin.keyring"
2021-02-13T20:47:09.526 INFO:scrape:4 jobs: ['5850873', '5851027', '5850972', '5850932']
2021-02-13T20:47:09.526 INFO:scrape:suites intersection: ['fixed-2', 'msgr/async-v1only', 'start']
2021-02-13T20:47:09.527 INFO:scrape:suites union: ['fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async-v1only', 'rados/cephadm/with-work/{distro/centos_8.0', 'rados/cephadm/with-work/{distro/centos_latest', 'rados/cephadm/with-work/{distro/rhel_8.0', 'rados/cephadm/with-work/{distro/rhel_latest', 'start', 'tasks/rados_api_tests}', 'tasks/rados_python}']
2021-02-13T20:47:09.527 INFO:scrape:
2021-02-13T20:47:09.527 INFO:scrape:Failure: Test failure: test_cluster_info (tasks.cephfs.test_nfs.TestNFS)
2021-02-13T20:47:09.527 INFO:scrape:2 jobs: ['5850891', '5850964']
2021-02-13T20:47:09.527 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{distro/centos_latest', 'task/test_orch_cli}']
2021-02-13T20:47:09.527 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'task/test_orch_cli}']
2021-02-13T20:47:09.527 INFO:scrape:
2021-02-13T20:47:09.527 INFO:scrape:Failure: Command failed (workunit test mgr/balancer.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c9fbc794811a1de66ee254a1f9078372839e4f3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mgr/balancer.sh'
2021-02-13T20:47:09.527 INFO:scrape:1 jobs: ['5850903']
2021-02-13T20:47:09.528 INFO:scrape:suites: ['rados/standalone/{mon_election/classic', 'supported-random-distro$/{rhel_8}', 'workloads/mgr}']
2021-02-13T20:47:09.528 INFO:scrape:
2021-02-13T20:47:09.528 INFO:scrape:Dead: b'2021-02-13T08:58:27.365 INFO:teuthology.orchestra.run.smithi091.stderr:'
2021-02-13T20:47:09.528 INFO:scrape:1 jobs: ['5851059']
2021-02-13T20:47:09.528 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}']
2021-02-13T20:47:09.528 INFO:scrape:
2021-02-13T20:47:09.528 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi194 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c9fbc794811a1de66ee254a1f9078372839e4f3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2021-02-13T20:47:09.528 INFO:scrape:1 jobs: ['5850926']
2021-02-13T20:47:09.528 INFO:scrape:suites: ['mon_election/classic', 'rados/cephadm/workunits/{distro/centos_latest', 'task/test_cephadm}']
2021-02-13T20:47:09.529 INFO:scrape:
2021-02-13T20:47:09.529 INFO:scrape:Failure: Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c9fbc794811a1de66ee254a1f9078372839e4f3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh'
2021-02-13T20:47:09.529 INFO:scrape:1 jobs: ['5850960']
2021-02-13T20:47:09.529 INFO:scrape:suites: ['rados/standalone/{mon_election/classic', 'supported-random-distro$/{rhel_8}', 'workloads/scrub}']
2021-02-13T20:47:09.529 INFO:scrape: