2020-12-11T10:03:05.402 INFO:root:teuthology version: 1.0.0-a4989a90
2020-12-11T10:03:05.403 INFO:teuthology.results:seed: '4745'
2020-12-11T10:22:51.569 INFO:root:teuthology version: 1.0.0-a4989a90
2020-12-11T10:22:51.610 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-12-11T22:21:04.780 INFO:teuthology.results:Tests finished! gathering results...
2020-12-11T22:21:11.025 INFO:scrape:Found 40 jobs
2020-12-11T22:21:21.232 INFO:scrape:Found 9 distinct failure reasons
2020-12-11T22:21:21.234 INFO:scrape:Failure: epo\n  ^ here\n", '_ansible_no_log': False}}Traceback (most recent call last):  File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure    log.error(yaml.safe_dump(failure))  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefinedRepresenterError: ('cannot represent an object', u"The conditional check 'perform_users_role|bool' failed. The error was: unable to evaluate conditional: perform_users_role|bool\n\nThe error appears to be in '/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/users/tasks/update_keys.yml': line 24, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Update authorized_keys using the keys repo\n  ^ here\n")
2020-12-11T22:21:21.235 INFO:scrape:10 jobs: ['5700873', '5700868', '5700864', '5700854', '5700856', '5700862', '5700855', '5700860', '5700870', '5700865']
2020-12-11T22:21:21.235 INFO:scrape:suites intersection: []
2020-12-11T22:21:21.236 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/3-mons', 'clusters/{fixed-2', 'd-require-luminous/at-end', 'd-require-luminous/at-mkfs-balancer-crush-compat', 'debug/mgr', 'fast/fast', 'leveldb', 'mon_kv_backend/leveldb', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async', 'msgr/random', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'openstack}', 'rados', 'rados/basic-luminous/{ceph', 'rados/mgr/{clusters/2-node-mgr', 'rados/monthrash/{ceph', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/singleton/{all/admin-socket', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code/{ceph', 'rados/thrash-luminous/{0-size-min-size-overrides/2-size-1-min-size', 'rados/thrash/{0-size-min-size-overrides/2-size-1-min-size', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados}', 'recovery-overrides/{more-partial-recovery}', 'rocksdb', 'scrub_test}', 'tasks/dashboard}', 'thrashers/default', 'thrashers/force-sync-many', 'thrashers/mapgap', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-agent-big}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/pool-create-delete}', 'workloads/redirect}']
2020-12-11T22:21:21.237 INFO:scrape:
2020-12-11T22:21:21.238 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op snap_remove 50 --op snap_create 50 --op rollback 50 --op read 100 --op copy_from 50 --op write 50 --op write_excl 50 --op cache_try_flush 50 --op cache_flush 50 --op cache_evict 50 --op delete 50 --pool base'
2020-12-11T22:21:21.239 INFO:scrape:14 jobs: ['5700877', '5700892', '5700889', '5700874', '5700886', '5700875', '5700883', '5700891', '5700890', '5700887', '5700888', '5700879', '5700884', '5700876']
2020-12-11T22:21:21.239 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'rocksdb', 'thrashosds-health']
2020-12-11T22:21:21.240 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'd-require-luminous/at-end', 'd-require-luminous/at-mkfs', 'd-require-luminous/at-mkfs-balancer-crush-compat', 'd-require-luminous/at-mkfs-balancer-upmap', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/random', 'msgr/simple', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-1-min-size', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rocksdb', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/pool-snaps-few-objects}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}']
2020-12-11T22:21:21.241 INFO:scrape:
2020-12-11T22:21:21.242 INFO:scrape:Assertion: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/12.2.13-671-g0c7c4abb/rpm/el7/BUILD/ceph-12.2.13-671-g0c7c4abb/src/mon/MonitorDBStore.h: 602: FAILED assert(0 == "MonitorDBStore: error initializing keyvaluedb back storage")
2020-12-11T22:21:21.243 INFO:scrape:ceph version 12.2.13-671-g0c7c4ab (0c7c4abb0348dff9f277beb9e8e9e2f3f499d571) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55f1c9c1cc30]
 2: (MonitorDBStore::_open(std::string)+0x507) [0x55f1c995fe37]
 3: (MonitorDBStore::open(std::ostream&)+0xaf) [0x55f1c99604ef]
 4: (main()+0xfeb) [0x55f1c98aeefb]
 5: (__libc_start_main()+0xf5) [0x7fa2bcb71445]
 6: (()+0x3dc1ee) [0x55f1c995c1ee]
2020-12-11T22:21:21.244 INFO:scrape:2 jobs: ['5700871', '5700867']
2020-12-11T22:21:21.244 INFO:scrape:suites intersection: []
2020-12-11T22:21:21.245 INFO:scrape:suites union: ['1-jewel-install/jewel', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-luminous', '8-workload/{rbd-python', 'ceph', 'clusters/{fixed-2', 'd-require-luminous/at-end', 'distros/centos_latest', 'leveldb', 'msgr-failures/fastclose', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/upgrade/jewel-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'recovery-overrides/{default}', 'rgw-swift', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported/centos_latest', 'thrashers/default', 'thrashosds-health', 'thrashosds-health}', 'workloads/ec-rados-plugin=isa-k=2-m=1}']
2020-12-11T22:21:21.246 INFO:scrape:
2020-12-11T22:21:21.247 INFO:scrape:Dead: b'2020-12-11T22:17:51.992 DEBUG:teuthology.exit:Finished running handlers'
2020-12-11T22:21:21.248 INFO:scrape:1 jobs: ['5700880']
2020-12-11T22:21:21.248 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'd-require-luminous/at-end', 'msgr-failures/osd-delay', 'msgr/simple', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rocksdb', 'thrashers/none', 'thrashosds-health', 'workloads/rados_api_tests}']
2020-12-11T22:21:21.249 INFO:scrape:
2020-12-11T22:21:21.250 INFO:scrape:Failure: ter.py", line 125, in represent_mapping  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefinedRepresenterError: ('cannot represent an object', u"The conditional check 'perform_users_role|bool' failed. The error was: unable to evaluate conditional: perform_users_role|bool\n\nThe error appears to be in '/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/users/tasks/create_users.yml': line 10, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Create all admin users with sudo access.\n  ^ here\n")
2020-12-11T22:21:21.251 INFO:scrape:2 jobs: ['5700861', '5700869']
2020-12-11T22:21:21.252 INFO:scrape:suites intersection: ['objectstore/bluestore-bitmap']
2020-12-11T22:21:21.252 INFO:scrape:suites union: ['clusters/{fixed-4', 'd-require-luminous/at-end', 'leveldb', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'rados/thrash-erasure-code-shec/{ceph', 'rados}', 'recovery-overrides/{more-active-recovery}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}']
2020-12-11T22:21:21.253 INFO:scrape:
2020-12-11T22:21:21.254 INFO:scrape:Failure: No module named pcp
2020-12-11T22:21:21.255 INFO:scrape:2 jobs: ['5700863', '5700858']
2020-12-11T22:21:21.255 INFO:scrape:suites intersection: []
2020-12-11T22:21:21.256 INFO:scrape:suites union: ['rados/objectstore/alloc-hint', 'rados/standalone/crush']
2020-12-11T22:21:21.257 INFO:scrape:
2020-12-11T22:21:21.257 INFO:scrape:Failure: {'smithi119.front.sepia.ceph.com': {'msg': 'Unexpected failure during module execution.', 'exception': 'Traceback (most recent call last):\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 110, in run\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 370, in _run_loop\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 648, in _execute\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/plugins/action/shell.py", line 27, in run\n    result = command_action.run(task_vars=task_vars)\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/plugins/action/command.py", line 24, in run\n    results = merge_hash(results, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 809, in _execute_module\n    (module_style, shebang, module_data, module_path) = self._configure_module(module_name=module_name, module_args=module_args, task_vars=task_vars)\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 203, in _configure_module\n    environment=final_environment)\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/executor/module_common.py", line 1015, in modify_module\nIOError: [Errno 2] No such file or directory: \'/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/modules/commands/command.py\'\n', '_ansible_no_log': False, 'stdout': ''}, 'smithi089.front.sepia.ceph.com': {'msg': 'Unexpected failure during module execution.', 'exception': 'Traceback (most recent call last):\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 110, in run\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 370, in _run_loop\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 648, in _execute\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/plugins/action/shell.py", line 27, in run\n    result = command_action.run(task_vars=task_vars)\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/plugins/action/command.py", line 24, in run\n    results = merge_hash(results, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 809, in _execute_module\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 203, in _configure_module\n  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/executor/module_common.py", line 1015, in modify_module\nIOError: [Errno 2] No such file or directory: \'/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/lib/python2.7/site-packages/ansible/modules/commands/command.py\'\n', '_ansible_no_log': False, 'stdout': ''}}
2020-12-11T22:21:21.258 INFO:scrape:5 jobs: ['5700853', '5700859', '5700857', '5700866', '5700872']
2020-12-11T22:21:21.259 INFO:scrape:suites intersection: []
2020-12-11T22:21:21.260 INFO:scrape:suites union: ['cluster/{12-osds', 'clusters/{fixed-2', 'd-require-luminous/at-end', 'd-thrash/default/{default', 'leveldb', 'mon_kv_backend/leveldb', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/multimon/{clusters/21', 'rados/rest/mgr-restful', 'rados/thrash-erasure-code-big/{ceph', 'rados/verify/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'tasks/mon_clock_no_skews}', 'tasks/mon_recovery', 'tasks/rados_api_tests}', 'thrashers/default', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}']
2020-12-11T22:21:21.260 INFO:scrape:
2020-12-11T22:21:21.261 INFO:scrape:Failure: [Errno 2] No such file or directory
2020-12-11T22:21:21.262 INFO:scrape:1 jobs: ['5700878']
2020-12-11T22:21:21.263 INFO:scrape:
2020-12-11T22:21:21.263 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op snap_remove 50 --op snap_create 50 --op rollback 50 --op setattr 25 --op read 100 --op copy_from 50 --op write 50 --op write_excl 50 --op rmattr 25 --op delete 50 --pool unique_pool_0'
2020-12-11T22:21:21.264 INFO:scrape:1 jobs: ['5700882']
2020-12-11T22:21:21.265 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'd-require-luminous/at-mkfs-balancer-upmap', 'msgr-failures/few', 'msgr/random', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rocksdb', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}']
2020-12-11T22:21:21.266 INFO:scrape: