2020-02-17T17:04:34.070 INFO:root:teuthology version: 1.0.0-7f4dc17 2020-02-17T17:04:34.070 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.wTnGWG.tmp -- --verbose --lock --description rados:monthrash/{ceph.yaml clusters/9-mons.yaml mon_election/connectivity.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_8.yaml} thrashers/many.yaml workloads/rados_mon_osdmap_prune.yaml} --name gregf-2020-02-17_05:30:48-rados:monthrash-wip-elector-squash-212-distro-basic-mira --block --owner scheduled_gregf@teuthology --archive /home/teuthworker/archive/gregf-2020-02-17_05:30:48-rados:monthrash-wip-elector-squash-212-distro-basic-mira/4774538 2020-02-17T17:04:34.103 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-17T17:04:34.176 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gregf-2020-02-17_05:30:48-rados:monthrash-wip-elector-squash-212-distro-basic-mira/4774538 branch: wip-elector-squash-212 description: rados:monthrash/{ceph.yaml clusters/9-mons.yaml mon_election/connectivity.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_8.yaml} thrashers/many.yaml workloads/rados_mon_osdmap_prune.yaml} email: gfarnum@redhat.com first_in_suite: false job_id: '4774538' kernel: kdb: true sha1: distro last_in_suite: false machine_type: mira name: gregf-2020-02-17_05:30:48-rados:monthrash-wip-elector-squash-212-distro-basic-mira nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.0' overrides: admin_socket: branch: wip-elector-squash-212 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug monc: 10 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon debug extra checks: true mon min osdmap epochs: 100 mon osdmap full prune enabled: true mon osdmap full prune interval: 10 mon osdmap full prune min: 200 mon osdmap full prune txsize: 100 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd beacon report interval: 10 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(OSDMAP_FLAGS\) sha1: 4e96f6d01c87d23d720091cfdaa0900b92d41fab ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4e96f6d01c87d23d720091cfdaa0900b92d41fab thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-elector-qa sha1: 620894dc894e35d055c06b84b6a83bb6db81b626 owner: scheduled_gregf@teuthology priority: 101 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 sha1: 4e96f6d01c87d23d720091cfdaa0900b92d41fab suite: rados:monthrash suite_branch: wip-elector-qa suite_path: /home/teuthworker/src/github.com_gregsfortytwo_ceph_wip-elector-qa/qa suite_relpath: qa suite_repo: https://github.com/gregsfortytwo/ceph.git suite_sha1: 620894dc894e35d055c06b84b6a83bb6db81b626 tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - mon/test_mon_osdmap_prune.sh teuthology_branch: master tube: mira verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.mira.10676 2020-02-17T17:04:34.176 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_gregsfortytwo_ceph_wip-elector-qa/qa; will attempt to use it 2020-02-17T17:04:34.177 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_gregsfortytwo_ceph_wip-elector-qa/qa/tasks 2020-02-17T17:04:34.178 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-02-17T17:04:34.180 INFO:teuthology.task.internal:Checking packages... 2020-02-17T17:04:34.201 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4e96f6d01c87d23d720091cfdaa0900b92d41fab' 2020-02-17T17:04:34.201 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-02-17T17:04:34.201 INFO:teuthology.packaging:ref: None 2020-02-17T17:04:34.201 INFO:teuthology.packaging:tag: None 2020-02-17T17:04:34.202 INFO:teuthology.packaging:branch: wip-elector-squash-212 2020-02-17T17:04:34.202 INFO:teuthology.packaging:sha1: 4e96f6d01c87d23d720091cfdaa0900b92d41fab 2020-02-17T17:04:34.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-elector-squash-212 2020-02-17T17:04:34.377 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-348.g4e96f6d 2020-02-17T17:04:34.378 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-02-17T17:04:34.394 INFO:teuthology.task.internal:no buildpackages task found 2020-02-17T17:04:34.394 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-02-17T17:04:34.417 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-02-17T17:04:34.418 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-17T17:04:34.475 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:04:44.493 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:04:54.511 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:05:04.531 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:05:14.547 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:05:24.565 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:05:34.582 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:05:44.602 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:05:54.620 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:06:04.630 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:06:14.647 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:06:24.664 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:06:34.683 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:06:44.697 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:06:54.715 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:07:04.733 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:07:14.751 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:07:24.766 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:07:34.784 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:07:44.803 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:07:54.819 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:08:04.836 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:08:14.848 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:08:24.864 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:08:34.880 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:08:44.903 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:08:54.920 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:09:04.936 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 2)... 2020-02-17T17:09:14.953 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 2, have 3)... 2020-02-17T17:09:24.971 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_gregf@teuthology', 'description': '/home/teuthworker/archive/gregf-2020-02-17_05:30:48-rados:monthrash-wip-elector-squash-212-distro-basic-mira/4774538', 'machine_type': 'mira'} 2020-02-17T17:09:24.991 DEBUG:teuthology.lock.ops:locked ubuntu@mira100.front.sepia.ceph.com, ubuntu@mira016.front.sepia.ceph.com 2020-02-17T17:09:25.057 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-02-17T17:09:25.319 INFO:teuthology.provision.fog.mira100:Scheduling deploy of centos 8.0 2020-02-17T17:09:25.374 INFO:teuthology.provision.fog.mira016:Scheduling deploy of centos 8.0 2020-02-17T17:09:25.870 INFO:teuthology.orchestra.console:Power off mira016 2020-02-17T17:09:25.870 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-02-17T17:09:25.884 INFO:teuthology.orchestra.console:Power off mira100 2020-02-17T17:09:25.885 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-02-17T17:09:25.906 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-17T17:09:25.914 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-17T17:09:26.031 INFO:teuthology.orchestra.console:Power off for mira016 completed 2020-02-17T17:09:26.040 INFO:teuthology.orchestra.console:Power off for mira100 completed 2020-02-17T17:09:26.134 INFO:teuthology.orchestra.console:Power on mira016 2020-02-17T17:09:26.134 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-02-17T17:09:26.146 INFO:teuthology.orchestra.console:Power on mira100 2020-02-17T17:09:26.147 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-02-17T17:09:26.161 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-17T17:09:26.170 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-17T17:09:30.189 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-17T17:09:30.200 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-17T17:09:34.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-17T17:09:34.311 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-17T17:09:34.513 INFO:teuthology.orchestra.console:Power on for mira016 completed 2020-02-17T17:09:34.520 INFO:teuthology.orchestra.console:Power on for mira100 completed 2020-02-17T17:09:34.614 INFO:teuthology.provision.fog.mira016:Waiting for deploy to finish 2020-02-17T17:09:34.622 INFO:teuthology.provision.fog.mira100:Waiting for deploy to finish 2020-02-17T17:12:24.592 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2020-02-17T17:12:24.593 DEBUG:teuthology.task.console_log:Killing console logger for mira016 2020-02-17T17:12:24.593 DEBUG:teuthology.task.console_log:Killing console logger for mira100 2020-02-17T17:12:24.593 DEBUG:teuthology.exit:Finished running handlers