2020-11-17T14:55:34.032 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-17T14:55:34.033 INFO:teuthology.results:subset: '16/100000' 2020-11-17T14:55:34.034 INFO:teuthology.results:seed: '3739' 2020-11-17T19:02:16.685 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-17T19:02:16.716 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-11-18T00:04:27.186 INFO:teuthology.results:Tests finished! gathering results... 2020-11-18T00:15:48.208 INFO:scrape:Found 366 jobs 2020-11-18T00:15:49.964 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7235-gff227345 (ff227345eb28743a41427f6d4cbc46f70082dda4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f553b964e94] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55c8ef8750b6] 2020-11-18T00:15:49.966 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:05:46.985 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-17T17:05:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: , dumped all 2020-11-18T00:15:49.966 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:06:49.209 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-11-17T17:06:55.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com , 2020-11-17T17:06:55.145 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-18T00:15:49.967 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:08:52.261 INFO:teuthology.task.internal:Duration was 667.122396 seconds , 2020-11-17T17:08:52.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2020-11-17T17:08:52.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-11-18T00:15:50.068 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7235-gff227345 (ff227345eb28743a41427f6d4cbc46f70082dda4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f553b964e94] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55c8ef8750b6] 2020-11-18T00:15:50.069 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:05:46.985 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-17T17:05:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: , dumped all 2020-11-18T00:15:50.070 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:06:49.209 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-11-17T17:06:55.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com , 2020-11-17T17:06:55.145 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-18T00:15:50.071 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:08:52.261 INFO:teuthology.task.internal:Duration was 667.122396 seconds , 2020-11-17T17:08:52.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2020-11-17T17:08:52.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-11-18T00:15:50.171 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7235-gff227345 (ff227345eb28743a41427f6d4cbc46f70082dda4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f553b964e94] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55c8ef8750b6] 2020-11-18T00:15:50.173 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:05:46.985 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-17T17:05:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: , dumped all 2020-11-18T00:15:50.173 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:06:49.209 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-11-17T17:06:55.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com , 2020-11-17T17:06:55.145 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-18T00:15:50.174 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:08:52.261 INFO:teuthology.task.internal:Duration was 667.122396 seconds , 2020-11-17T17:08:52.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2020-11-17T17:08:52.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-11-18T00:15:50.276 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7235-gff227345 (ff227345eb28743a41427f6d4cbc46f70082dda4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f553b964e94] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55c8ef8750b6] 2020-11-18T00:15:50.278 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:05:46.985 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-17T17:05:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: , dumped all 2020-11-18T00:15:50.279 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:06:49.209 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-11-17T17:06:55.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com , 2020-11-17T17:06:55.145 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-18T00:15:50.279 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:08:52.261 INFO:teuthology.task.internal:Duration was 667.122396 seconds , 2020-11-17T17:08:52.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2020-11-17T17:08:52.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-11-18T00:15:50.385 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7235-gff227345 (ff227345eb28743a41427f6d4cbc46f70082dda4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f553b964e94] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55c8ef8750b6] 2020-11-18T00:15:50.387 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:05:46.985 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-17T17:05:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: , dumped all 2020-11-18T00:15:50.387 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:06:49.209 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-11-17T17:06:55.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com , 2020-11-17T17:06:55.145 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-18T00:15:50.388 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:08:52.261 INFO:teuthology.task.internal:Duration was 667.122396 seconds , 2020-11-17T17:08:52.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2020-11-17T17:08:52.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-11-18T00:15:50.490 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7235-gff227345 (ff227345eb28743a41427f6d4cbc46f70082dda4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f553b964e94] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55c8ef8750b6] 2020-11-18T00:15:50.492 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:05:46.985 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-17T17:05:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: , dumped all 2020-11-18T00:15:50.492 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:06:49.209 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-11-17T17:06:55.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com , 2020-11-17T17:06:55.145 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-18T00:15:50.493 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:08:52.261 INFO:teuthology.task.internal:Duration was 667.122396 seconds , 2020-11-17T17:08:52.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2020-11-17T17:08:52.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-11-18T00:15:50.592 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7235-gff227345 (ff227345eb28743a41427f6d4cbc46f70082dda4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f553b964e94] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55c8ef8750b6] 2020-11-18T00:15:50.594 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:05:46.985 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-17T17:05:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: , dumped all 2020-11-18T00:15:50.595 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:06:49.209 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-11-17T17:06:55.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com , 2020-11-17T17:06:55.145 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-18T00:15:50.595 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:08:52.261 INFO:teuthology.task.internal:Duration was 667.122396 seconds , 2020-11-17T17:08:52.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2020-11-17T17:08:52.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-11-18T00:15:50.736 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7235-gff227345 (ff227345eb28743a41427f6d4cbc46f70082dda4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f553b964e94] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55c8ef8750b6] 2020-11-18T00:15:50.739 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:05:46.985 INFO:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-11-17T17:05:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: , dumped all 2020-11-18T00:15:50.740 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:06:49.209 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-11-17T17:06:55.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com , 2020-11-17T17:06:55.145 INFO:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-18T00:15:50.741 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-17T17:08:52.261 INFO:teuthology.task.internal:Duration was 667.122396 seconds , 2020-11-17T17:08:52.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2020-11-17T17:08:52.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-11-18T00:16:00.374 INFO:scrape:Found 15 distinct failure reasons 2020-11-18T00:16:00.375 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2020-11-18T00:16:00.375 INFO:scrape:3 jobs: ['5633087', '5633220', '5633359'] 2020-11-18T00:16:00.376 INFO:scrape:suites intersection: ['task/test_adoption}'] 2020-11-18T00:16:00.376 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2020-11-18T00:16:00.377 INFO:scrape: 2020-11-18T00:16:00.377 INFO:scrape:Failure: "2020-11-17T18:59:59.828355+0000 mon.b (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log 2020-11-18T00:16:00.378 INFO:scrape:1 jobs: ['5633319'] 2020-11-18T00:16:00.378 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] 2020-11-18T00:16:00.378 INFO:scrape: 2020-11-18T00:16:00.379 INFO:scrape:Failure: Command failed on smithi158 with status 5: 'sudo systemctl stop ceph-18a59fbe-28f1-11eb-b4fd-001a4aab830c@mon.a' 2020-11-18T00:16:00.379 INFO:scrape:8 jobs: ['5633090', '5633235', '5633270', '5633363', '5633318', '5633175', '5633132', '5633052'] 2020-11-18T00:16:00.380 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] 2020-11-18T00:16:00.380 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_8.0}', 'distro$/{centos_latest}', 'distro$/{rhel_8.0}', 'distro$/{ubuntu_18.04_podman}', 'distro$/{ubuntu_18.04}', 'distro$/{ubuntu_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-11-18T00:16:00.381 INFO:scrape: 2020-11-18T00:16:00.381 INFO:scrape:Failure: Command failed (workunit test misc/ver-health.sh) on smithi085 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' 2020-11-18T00:16:00.381 INFO:scrape:2 jobs: ['5633291', '5633107'] 2020-11-18T00:16:00.382 INFO:scrape:suites intersection: ['workloads/misc}'] 2020-11-18T00:16:00.382 INFO:scrape:suites union: ['rados/standalone/{mon_election/classic', 'rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/misc}'] 2020-11-18T00:16:00.383 INFO:scrape: 2020-11-18T00:16:00.383 INFO:scrape:Failure: ceph version 14.2.14-1bionic was not installed, found 14.2.13-58-gb0dfafeb-1bionic. 2020-11-18T00:16:00.383 INFO:scrape:1 jobs: ['5633338'] 2020-11-18T00:16:00.384 INFO:scrape:suites: ['1-install/nautilus-v1only', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'distro$/{ubuntu_18.04}', 'mon_election/classic', 'msgr-failures/few', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/snaps-few-objects}'] 2020-11-18T00:16:00.384 INFO:scrape: 2020-11-18T00:16:00.385 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-11-18T00:16:00.385 INFO:scrape:1 jobs: ['5633140'] 2020-11-18T00:16:00.385 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{centos_8}}'] 2020-11-18T00:16:00.386 INFO:scrape: 2020-11-18T00:16:00.386 INFO:scrape:Failure: HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/master/ff227345eb28743a41427f6d4cbc46f70082dda4/centos/8/flavors/default/repo (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),)) 2020-11-18T00:16:00.386 INFO:scrape:1 jobs: ['5633280'] 2020-11-18T00:16:00.387 INFO:scrape:suites: ['cluster/{12-osds', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}'] 2020-11-18T00:16:00.387 INFO:scrape: 2020-11-18T00:16:00.388 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi085 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2020-11-18T00:16:00.388 INFO:scrape:2 jobs: ['5633125', '5633258'] 2020-11-18T00:16:00.388 INFO:scrape:suites intersection: ['task/test_cephadm}'] 2020-11-18T00:16:00.389 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-11-18T00:16:00.389 INFO:scrape: 2020-11-18T00:16:00.390 INFO:scrape:Failure: timeout expired in wait_until_healthy 2020-11-18T00:16:00.390 INFO:scrape:1 jobs: ['5633168'] 2020-11-18T00:16:00.390 INFO:scrape:suites: ['rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman', 'start}'] 2020-11-18T00:16:00.391 INFO:scrape: 2020-11-18T00:16:00.391 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/mon/caps.sh 2020-11-18T00:16:00.392 INFO:scrape:1 jobs: ['5633223'] 2020-11-18T00:16:00.392 INFO:scrape:suites: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{rhel_8}', 'thrashers/many', 'workloads/rados_mon_workunits}'] 2020-11-18T00:16:00.392 INFO:scrape: 2020-11-18T00:16:00.393 INFO:scrape:Failure: Command failed (workunit test mon/mon-last-epoch-clean.sh) on smithi076 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-last-epoch-clean.sh' 2020-11-18T00:16:00.393 INFO:scrape:1 jobs: ['5633316'] 2020-11-18T00:16:00.394 INFO:scrape:suites: ['rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{rhel_8}', 'workloads/mon}'] 2020-11-18T00:16:00.394 INFO:scrape: 2020-11-18T00:16:00.394 INFO:scrape:Failure: Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh' 2020-11-18T00:16:00.395 INFO:scrape:1 jobs: ['5633186'] 2020-11-18T00:16:00.395 INFO:scrape:suites: ['rados/standalone/{mon_election/classic', 'supported-random-distro$/{centos_8}', 'workloads/scrub}'] 2020-11-18T00:16:00.395 INFO:scrape: 2020-11-18T00:16:00.396 INFO:scrape:Failure: failed to reach quorum size 9 before timeout expired 2020-11-18T00:16:00.396 INFO:scrape:1 jobs: ['5633185'] 2020-11-18T00:16:00.397 INFO:scrape:suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/force-sync-many', 'workloads/rados_mon_osdmap_prune}'] 2020-11-18T00:16:00.397 INFO:scrape: 2020-11-18T00:16:00.397 INFO:scrape:Crash: "2020-11-17T17:32:46.572361+0000 mon.b (mon.0) 51 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log 2020-11-18T00:16:00.398 INFO:scrape:ceph version 16.0.0-7235-gff227345 (ff227345eb28743a41427f6d4cbc46f70082dda4) pacific (dev) 1: /lib64/libpthread.so.0(+0x12dc0) [0xf9a5dc0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x50b5221] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2773ea) [0x50b53ea] 6: (Paxos::commit_proposal()+0x16d) [0x4f2d1d] 7: (Paxos::commit_finish()+0x898) [0x4fbbb8] 8: (C_Committed::finish(int)+0x38) [0x500008] 9: (Context::complete(int)+0xd) [0x3f278d] 10: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x4ffd38] 11: (Context::complete(int)+0xd) [0x3f278d] 12: (Finisher::finisher_thread_entry()+0x28d) [0x515868d] 13: /lib64/libpthread.so.0(+0x82de) [0xf99b2de] 14: clone() 2020-11-18T00:16:00.398 INFO:scrape:1 jobs: ['5633134'] 2020-11-18T00:16:00.399 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'validater/valgrind}'] 2020-11-18T00:16:00.399 INFO:scrape: 2020-11-18T00:16:00.399 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2020-11-18T00:16:00.400 INFO:scrape:1 jobs: ['5633283'] 2020-11-18T00:16:00.400 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] 2020-11-18T00:16:00.401 INFO:scrape: 2020-11-18T00:16:00.412 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/ failed: 26 dead: 0 running: 0 waiting: 0 queued: 0 passed: 340 Fail ================================================================= [5633052] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:02:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633052/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633052/ sentry: https://sentry.ceph.com/organizations/ceph/?query=212e4ad3e70b414da32da443cfe530f0 Command failed on smithi109 with status 5: 'sudo systemctl stop ceph- ecb2d742-28ea-11eb-b4fd-001a4aab830c@mon.a' [5633087] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_adoption} ----------------------------------------------------------------- time: 00:10:49 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633087/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633087/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64ac577f469649a7bd10c0a9cec5f3ee Command failed (workunit test cephadm/test_adoption.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5633090] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:04:18 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633090/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633090/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aeecdb286c054ee38dfeb024c5ff6102 Command failed on smithi158 with status 5: 'sudo systemctl stop ceph-18a59fbe-28f1-11eb-b4fd-001a4aab830c@mon.a' [5633107] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/misc} ----------------------------------------------------------------- time: 00:34:40 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633107/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633107/ sentry: https://sentry.ceph.com/organizations/ceph/?query=089062e3d6a2436baf97d7f06be8375e Command failed (workunit test misc/ver-health.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' [5633125] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:09:16 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633125/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633125/ sentry: https://sentry.ceph.com/organizations/ceph/?query=44e15022752c41bc8448335d5c67d748 Command failed (workunit test cephadm/test_cephadm.sh) on smithi085 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5633132] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:07:28 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633132/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633132/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8a438462f814fe3a3ed81b56606126e Command failed on smithi001 with status 5: 'sudo systemctl stop ceph-f7f6a1fe-28f6-11eb-b4fd-001a4aab830c@mon.a' [5633134] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:27:58 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633134/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633134/ "2020-11-17T17:32:46.572361+0000 mon.b (mon.0) 51 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log [5633140] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:07 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633140/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633140/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5633168] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} ----------------------------------------------------------------- time: 00:31:43 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633168/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633168/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7df487fdc25345669cbaf32e7511ddf4 timeout expired in wait_until_healthy [5633175] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:02:42 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633175/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633175/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94d120dc162c42529350df7306f0f3ee Command failed on smithi131 with status 5: 'sudo systemctl stop ceph-f81f1af0-28f9-11eb-b4fd-001a4aab830c@mon.a' [5633185] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:44:55 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633185/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633185/ failed to reach quorum size 9 before timeout expired [5633186] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/scrub} ----------------------------------------------------------------- time: 00:21:42 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633186/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9abb5dcb63e140709a2f66dd69bc58f2 Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery- scrub.sh' [5633220] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:09:31 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633220/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633220/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19e5b2460af247e8a26c2b16193905e9 Command failed (workunit test cephadm/test_adoption.sh) on smithi001 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5633223] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 03:25:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633223/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633223/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5a3ecb1086174e83baaa2b7419ee7a1b Command failed (workunit test mon/caps.sh) on smithi109 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh' [5633235] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:02:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633235/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633235/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07a9b5376bb94192a616f7744bf5e7ec Command failed on smithi073 with status 5: 'sudo systemctl stop ceph-41753c34-28ff-11eb-b4fd-001a4aab830c@mon.a' [5633258] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:10:51 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633258/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633258/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f310d862b2a640aa8431d6b63d8338d2 Command failed (workunit test cephadm/test_cephadm.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5633270] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:02:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633270/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633270/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a97bcb570e0647998efdbe68b331323f Command failed on smithi025 with status 5: 'sudo systemctl stop ceph-475c4ec8-2902-11eb-b4fd-001a4aab830c@mon.a' [5633280] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:04:23 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633280/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633280/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03f2ce92586a43a18098837a740dba2f HTTPSConnectionPool(host='4.chacra.ceph.com', port=443): Max retries exceeded with url: /repos/ceph/master/ff227345eb28743a41427f6d4cbc46f70082d da4/centos/8/flavors/default/repo (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')",),)) [5633283] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:40:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633283/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633283/ Command failed on smithi065 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [5633291] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/misc} ----------------------------------------------------------------- time: 00:28:43 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633291/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633291/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6bdaa9c1a60422a88d976bb09930b71 Command failed (workunit test misc/ver-health.sh) on smithi085 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' [5633316] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mon} ----------------------------------------------------------------- time: 00:26:20 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633316/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633316/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a54dd5b5fd24b71807f48e8587c51d1 Command failed (workunit test mon/mon-last-epoch-clean.sh) on smithi076 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-last-epoch- clean.sh' [5633318] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:02:05 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633318/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633318/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3ccd10e5a1a4d83af01ee940345a689 Command failed on smithi199 with status 5: 'sudo systemctl stop ceph-1a4ae224-2906-11eb-b4fd-001a4aab830c@mon.a' [5633319] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:28:09 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633319/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633319/ "2020-11-17T18:59:59.828355+0000 mon.b (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log [5633338] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:03:53 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633338/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633338/ sentry: https://sentry.ceph.com/organizations/ceph/?query=afd007d321f94ee6b902eda8854d8bf1 ceph version 14.2.14-1bionic was not installed, found 14.2.13-58-gb0dfafeb-1bionic. [5633359] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} ----------------------------------------------------------------- time: 00:09:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633359/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633359/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0db306810360414a9cc795e8b71db751 Command failed (workunit test cephadm/test_adoption.sh) on smithi205 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ff227345eb28743a41427f6d4cbc46f70082dda4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5633363] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:02:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633363/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633363/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a47094872b14551b79edc7ff21ef130 Command failed on smithi155 with status 5: 'sudo systemctl stop ceph-64fa36a6-2908-11eb-b4fd-001a4aab830c@mon.a' Pass ================================================================= [5633004] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache} time: 00:09:22 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633004/ [5633005] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python} time: 00:24:38 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633005/ [5633006] rados/singleton/{all/peer mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:05:13 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633006/ [5633007] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:24:46 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633007/ [5633008] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:26:08 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633008/ [5633009] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/dashboard} time: 02:09:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633009/ [5633010] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:26 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633010/ [5633011] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:12:22 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633011/ [5633012] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:11:20 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633012/ [5633013] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:33:31 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633013/ [5633014] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm_repos} time: 00:07:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633014/ [5633015] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:28 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633015/ [5633016] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:16:09 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633016/ [5633017] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:37:09 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633017/ [5633018] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:12:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633018/ [5633019] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:11:24 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633019/ [5633020] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:05:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633020/ [5633021] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:55 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633021/ [5633022] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/lockdep} time: 00:10:10 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633022/ [5633023] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:31:25 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633023/ [5633024] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:41:26 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633024/ [5633025] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/crash} time: 00:09:50 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633025/ [5633026] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:44:31 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633026/ [5633027] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:30:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633027/ [5633028] rados/objectstore/{backends/alloc-hint supported-random-distro$/{rhel_8}} time: 00:16:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633028/ [5633029] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:11:52 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633029/ [5633030] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:19:27 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633030/ [5633031] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:11:59 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633031/ [5633032] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:40:12 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633032/ [5633033] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} time: 04:53:30 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633033/ [5633034] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:15:52 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633034/ [5633035] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:05:08 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633035/ [5633036] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:44:58 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633036/ [5633037] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:56:24 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633037/ [5633038] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:59 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633038/ [5633039] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:22:12 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633039/ [5633040] rados/singleton/{all/radostool mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:18:16 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633040/ [5633041] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 01:02:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633041/ [5633042] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{rhel_8}} time: 00:16:23 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633042/ [5633043] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:31 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633043/ [5633044] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_mon_osdmap_prune} time: 01:00:38 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633044/ [5633045] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:12:05 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633045/ [5633046] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:14:07 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633046/ [5633047] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:10:26 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633047/ [5633048] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:29:32 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633048/ [5633049] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/dashboard} time: 01:48:22 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633049/ [5633050] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:05:53 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633050/ [5633051] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/radosbench} time: 01:39:24 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633051/ [5633053] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix} time: 00:28:34 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633053/ [5633054] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:16:07 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633054/ [5633055] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 00:37:45 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633055/ [5633056] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:19:14 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633056/ [5633057] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:44:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633057/ [5633058] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:10:45 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633058/ [5633059] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633059/ [5633060] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect} time: 00:19:12 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633060/ [5633061] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:29:26 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633061/ [5633062] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:35:26 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633062/ [5633063] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} time: 00:47:55 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633063/ [5633064] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:10:14 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633064/ [5633065] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:16 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633065/ [5633066] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:19:32 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633066/ [5633067] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:22:57 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633067/ [5633068] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:09:59 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633068/ [5633069] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:42:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633069/ [5633070] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/dashboard} time: 01:26:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633070/ [5633071] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{rhel_8}} time: 00:14:49 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633071/ [5633072] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:37 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633072/ [5633073] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:05 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633073/ [5633074] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:16:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633074/ [5633075] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:08:07 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633075/ [5633076] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:20:53 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633076/ [5633077] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633077/ [5633078] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/one workloads/pool-create-delete} time: 00:15:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633078/ [5633079] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:31:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633079/ [5633080] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:05:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633080/ [5633081] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:14:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633081/ [5633082] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/mgr} time: 00:12:18 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633082/ [5633083] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:11:22 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633083/ [5633084] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:37:26 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633084/ [5633085] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:35:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633085/ [5633086] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:11:39 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633086/ [5633088] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:11:10 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633088/ [5633089] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633089/ [5633091] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:11:57 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633091/ [5633092] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:05:48 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633092/ [5633093] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633093/ [5633094] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:17:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633094/ [5633095] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/insights} time: 00:15:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633095/ [5633096] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:29 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633096/ [5633097] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/lockdep} time: 00:21:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633097/ [5633098] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:16:08 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633098/ [5633099] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:16:55 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633099/ [5633100] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633100/ [5633101] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:19 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633101/ [5633102] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:41:15 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633102/ [5633103] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633103/ [5633104] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:05:22 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633104/ [5633105] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-radosbench} time: 02:33:16 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633105/ [5633106] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:13:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633106/ [5633108] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} time: 00:20:06 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633108/ [5633109] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 01:09:11 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633109/ [5633110] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:13:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633110/ [5633111] rados/objectstore/{backends/filejournal supported-random-distro$/{rhel_8}} time: 00:10:40 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633111/ [5633112] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:20:44 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633112/ [5633113] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:25:13 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633113/ [5633114] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_5925} time: 00:08:08 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633114/ [5633115] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:12:57 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633115/ [5633116] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633116/ [5633117] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/readwrite} time: 00:15:16 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633117/ [5633118] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:08:10 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633118/ [5633119] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:16:26 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633119/ [5633120] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:27:14 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633120/ [5633121] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:36:09 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633121/ [5633122] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:44:33 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633122/ [5633123] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:22:05 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633123/ [5633124] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:07:13 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633124/ [5633126] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:11:52 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633126/ [5633127] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects} time: 00:26:00 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633127/ [5633128] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:21:07 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633128/ [5633129] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:21:38 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633129/ [5633130] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:26:53 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633130/ [5633131] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/mon} time: 00:52:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633131/ [5633133] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:31:56 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633133/ [5633135] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:59 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633135/ [5633136] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:19 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633136/ [5633137] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:36 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633137/ [5633138] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:24:07 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633138/ [5633139] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:10:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633139/ [5633141] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:29:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633141/ [5633142] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:12:48 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633142/ [5633143] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-balanced} time: 00:19:12 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633143/ [5633144] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} time: 00:23:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633144/ [5633145] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:10:14 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633145/ [5633146] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/repair_test} time: 00:13:46 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633146/ [5633147] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:55 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633147/ [5633148] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:10:53 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633148/ [5633149] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_api_tests} time: 00:24:46 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633149/ [5633150] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:19:31 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633150/ [5633151] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{rhel_8}} time: 02:21:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633151/ [5633152] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:17:56 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633152/ [5633153] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:05:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633153/ [5633154] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm_repos} time: 00:07:15 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633154/ [5633155] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:33:52 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633155/ [5633156] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:32:10 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633156/ [5633157] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:19:20 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633157/ [5633158] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/osd} time: 03:12:49 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633158/ [5633159] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:14:55 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633159/ [5633160] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:08:40 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633160/ [5633161] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:04:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633161/ [5633162] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633162/ [5633163] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:21:32 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633163/ [5633164] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:20:18 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633164/ [5633165] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:14:32 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633165/ [5633166] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/progress} time: 00:14:36 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633166/ [5633167] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:15:21 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633167/ [5633169] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:07:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633169/ [5633170] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:07 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633170/ [5633171] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:14:10 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633171/ [5633172] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} time: 00:24:56 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633172/ [5633173] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:14:58 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633173/ [5633174] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:20:59 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633174/ [5633176] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:29:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633176/ [5633177] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/scrub_test} time: 00:10:10 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633177/ [5633178] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:10:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633178/ [5633179] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read} time: 00:19:45 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633179/ [5633180] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:33:07 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633180/ [5633181] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:31 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633181/ [5633182] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:09:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633182/ [5633183] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} time: 00:30:46 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633183/ [5633184] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:10:27 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633184/ [5633187] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:14:17 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633187/ [5633188] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests} time: 00:40:53 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633188/ [5633189] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:19:23 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633189/ [5633190] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:32:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633190/ [5633191] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:13:40 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633191/ [5633192] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8}} time: 02:31:49 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633192/ [5633193] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:13:48 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633193/ [5633194] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:12:53 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633194/ [5633195] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633195/ [5633196] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-big} time: 00:17:10 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633196/ [5633197] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:47 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633197/ [5633198] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:53:27 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633198/ [5633199] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/prometheus} time: 00:08:49 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633199/ [5633200] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:13:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633200/ [5633201] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:38:18 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633201/ [5633202] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:09 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633202/ [5633203] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/rados_api_tests} time: 00:35:59 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633203/ [5633204] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:08:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633204/ [5633205] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633205/ [5633206] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} time: 02:16:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633206/ [5633207] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:13:11 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633207/ [5633208] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633208/ [5633209] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small} time: 00:16:57 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633209/ [5633210] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:35:32 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633210/ [5633211] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:11:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633211/ [5633212] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:13:32 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633212/ [5633213] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:11:46 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633213/ [5633214] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} time: 05:26:19 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633214/ [5633215] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:21:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633215/ [5633216] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:48:20 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633216/ [5633217] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:04:40 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633217/ [5633218] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:25:36 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633218/ [5633219] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:25:16 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633219/ [5633221] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:21:34 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633221/ [5633222] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:21:23 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633222/ [5633224] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633224/ [5633225] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:09:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633225/ [5633226] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}} time: 00:10:40 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633226/ [5633227] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:12:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633227/ [5633228] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:07:52 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633228/ [5633229] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:14:11 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633229/ [5633230] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:10:19 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633230/ [5633231] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:08:37 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633231/ [5633232] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-pool-snaps} time: 00:24:50 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633232/ [5633233] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:11:56 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633233/ [5633234] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:44:58 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633234/ [5633236] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/rados_cls_all} time: 00:23:34 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633236/ [5633237] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:09:48 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633237/ [5633238] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/erasure-code} time: 00:39:06 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633238/ [5633239] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:24:50 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633239/ [5633240] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:08:12 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633240/ [5633241] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:10:00 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633241/ [5633242] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:29 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633242/ [5633243] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:09:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633243/ [5633244] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:24:03 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633244/ [5633245] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:10:16 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633245/ [5633246] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:47:51 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633246/ [5633247] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:26 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633247/ [5633248] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:37:15 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633248/ [5633249] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:21 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633249/ [5633250] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:05:31 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633250/ [5633251] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects} time: 00:18:00 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633251/ [5633252] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:21:04 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633252/ [5633253] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633253/ [5633254] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:11:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633254/ [5633255] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:04:55 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633255/ [5633256] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:48 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633256/ [5633257] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:20:37 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633257/ [5633259] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} time: 00:33:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633259/ [5633260] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:15:17 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633260/ [5633261] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/mgr} time: 00:12:30 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633261/ [5633262] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:35:32 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633262/ [5633263] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:15:49 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633263/ [5633264] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:05:38 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633264/ [5633265] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:27:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633265/ [5633266] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache} time: 00:11:30 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633266/ [5633267] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/rados_python} time: 00:15:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633267/ [5633268] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:14:27 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633268/ [5633269] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/crash} time: 00:13:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633269/ [5633271] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} time: 01:05:18 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633271/ [5633272] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{rhel_8}} time: 00:16:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633272/ [5633273] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:27 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633273/ [5633274] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:17:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633274/ [5633275] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:10:32 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633275/ [5633276] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:16:25 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633276/ [5633277] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:53 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633277/ [5633278] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:05:30 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633278/ [5633279] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:23:37 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633279/ [5633281] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} time: 00:46:36 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633281/ [5633282] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:16:45 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633282/ [5633284] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:30:59 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633284/ [5633285] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:10:48 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633285/ [5633286] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:19:51 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633286/ [5633287] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:31:06 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633287/ [5633288] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:31 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633288/ [5633289] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} time: 00:24:09 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633289/ [5633290] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:29:44 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633290/ [5633292] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:12:56 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633292/ [5633293] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:27 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633293/ [5633294] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/pool-create-delete} time: 00:19:40 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633294/ [5633295] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:07:07 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633295/ [5633296] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:17:47 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633296/ [5633297] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:08:13 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633297/ [5633298] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:18:25 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633298/ [5633299] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:15 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633299/ [5633300] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:24:54 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633300/ [5633301] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:36:00 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633301/ [5633302] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:12:49 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633302/ [5633303] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:36:44 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633303/ [5633304] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:16:22 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633304/ [5633305] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:15:13 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633305/ [5633306] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:17 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633306/ [5633307] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/failover} time: 00:15:31 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633307/ [5633308] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:11:42 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633308/ [5633309] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:32:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633309/ [5633310] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench} time: 02:17:48 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633310/ [5633311] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:01 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633311/ [5633312] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{rhel_8}} time: 00:29:45 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633312/ [5633313] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:12:15 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633313/ [5633314] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:27 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633314/ [5633315] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/filestore-xfs rados tasks/rados_cls_all validater/lockdep} time: 00:18:20 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633315/ [5633317] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:10:29 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633317/ [5633320] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:28:11 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633320/ [5633321] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:10:08 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633321/ [5633322] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:14:58 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633322/ [5633323] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:23:11 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633323/ [5633324] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:10:41 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633324/ [5633325] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:29:14 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633325/ [5633326] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect} time: 00:17:20 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633326/ [5633327] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_orch_cli} time: 00:16:14 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633327/ [5633328] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:17:12 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633328/ [5633329] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_striper} time: 00:11:39 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633329/ [5633330] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:10:51 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633330/ [5633331] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:48 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633331/ [5633332] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_5925} time: 00:13:35 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633332/ [5633333] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:58 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633333/ [5633334] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:12:53 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633334/ [5633335] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:16:47 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633335/ [5633336] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:30:49 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633336/ [5633337] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:17:26 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633337/ [5633339] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:56 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633339/ [5633340] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:25:52 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633340/ [5633341] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:10:08 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633341/ [5633342] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/osd} time: 03:07:12 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633342/ [5633343] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:20:25 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633343/ [5633344] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:16:09 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633344/ [5633345] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:11:23 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633345/ [5633346] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:16:49 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633346/ [5633347] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:11:43 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633347/ [5633348] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:19:19 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633348/ [5633349] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:21:28 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633349/ [5633350] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:51 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633350/ [5633351] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados tasks/rados_cls_all validater/valgrind} time: 02:04:18 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633351/ [5633352] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:08:02 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633352/ [5633353] rados/objectstore/{backends/objectstore supported-random-distro$/{centos_8}} time: 04:56:47 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633353/ [5633354] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:47 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633354/ [5633355] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:20:10 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633355/ [5633356] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:14:30 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633356/ [5633357] rados/singleton/{all/peer mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:11:23 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633357/ [5633358] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_big} time: 00:32:50 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633358/ [5633360] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:05:28 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633360/ [5633361] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:23:32 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633361/ [5633362] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:47 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633362/ [5633364] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:09:57 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633364/ [5633365] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:38:27 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633365/ [5633366] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_api_tests} time: 00:30:34 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633366/ [5633367] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/scrub} time: 01:39:52 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633367/ [5633368] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:12:00 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633368/ [5633369] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:21:59 info: http://pulpito.ceph.com/teuthology-2020-11-17_07:01:02-rados-master-distro-basic-smithi/5633369/