2020-01-16T00:40:07.614 INFO:root:teuthology version: 1.0.0-7a27738 2020-01-16T00:40:07.615 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.QxR98o.tmp -- --verbose --lock --description rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{rhel_8.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} --name sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi --block --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671750 2020-01-16T00:40:07.655 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-01-16T00:40:07.723 INFO:teuthology.run:Config: arch: x86_64 archive_path: /home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671750 branch: wip-sage-testing-2020-01-15-1209 description: rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{rhel_8.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} email: null first_in_suite: false job_id: '4671750' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: rhel os_version: '8.0' overrides: admin_socket: branch: wip-sage-testing-2020-01-15-1209 ceph: conf: global: mon client directed command retry: 5 ms inject socket failures: 2500 ms tcp read timeout: 5 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow requests sha1: cb3c85e74500283e01cc592f7e2858e323186910 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: cb3c85e74500283e01cc592f7e2858e323186910 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-sage-testing-2020-01-15-1209 sha1: cb3c85e74500283e01cc592f7e2858e323186910 owner: scheduled_sage@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: cb3c85e74500283e01cc592f7e2858e323186910 suite: rados suite_branch: wip-sage-testing-2020-01-15-1209 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: cb3c85e74500283e01cc592f7e2858e323186910 tasks: - install: null - ceph: null - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: crush-failure-domain: osd k: 2 m: 1 name: isaprofile plugin: isa technique: reed_sol_van min_size: 2 objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 4000 write_append_excl: false teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.17173 2020-01-16T00:40:07.723 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa; will attempt to use it 2020-01-16T00:40:07.724 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks 2020-01-16T00:40:07.724 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-01-16T00:40:07.893 INFO:teuthology.task.internal:Checking packages... 2020-01-16T00:40:08.206 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'cb3c85e74500283e01cc592f7e2858e323186910' 2020-01-16T00:40:08.206 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-01-16T00:40:08.206 INFO:teuthology.packaging:ref: None 2020-01-16T00:40:08.206 INFO:teuthology.packaging:tag: None 2020-01-16T00:40:08.206 INFO:teuthology.packaging:branch: wip-sage-testing-2020-01-15-1209 2020-01-16T00:40:08.207 INFO:teuthology.packaging:sha1: cb3c85e74500283e01cc592f7e2858e323186910 2020-01-16T00:40:08.207 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-sage-testing-2020-01-15-1209 2020-01-16T00:40:08.362 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-9353.gcb3c85e 2020-01-16T00:40:08.362 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-01-16T00:40:08.371 INFO:teuthology.task.internal:no buildpackages task found 2020-01-16T00:40:08.371 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-01-16T00:40:08.385 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-01-16T00:40:08.385 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-01-16T00:40:08.432 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:40:18.449 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:40:28.468 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:40:38.485 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:40:48.503 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:40:58.521 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:41:08.538 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:41:18.556 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:41:28.574 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:41:38.590 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-01-16T00:41:48.609 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-01-16T00:41:58.627 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-01-16T00:42:08.642 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-01-16T00:42:18.657 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-01-16T00:42:28.684 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-01-16T00:42:38.701 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-01-16T00:42:48.719 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-01-16T00:42:58.736 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-01-16T00:43:08.755 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:43:18.773 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:43:28.791 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:43:38.811 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:43:48.827 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:43:58.846 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:44:08.862 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:44:18.878 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:44:28.896 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:44:38.914 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:44:48.933 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:44:58.946 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-01-16T00:45:08.962 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_sage@teuthology', 'arch': 'x86_64', 'description': '/home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671750', 'machine_type': 'smithi'} 2020-01-16T00:45:08.982 DEBUG:teuthology.lock.ops:locked ubuntu@smithi130.front.sepia.ceph.com, ubuntu@smithi086.front.sepia.ceph.com 2020-01-16T00:45:09.041 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-01-16T00:45:09.251 INFO:teuthology.provision.fog.smithi086:Scheduling deploy of rhel 8.0 2020-01-16T00:45:09.257 INFO:teuthology.provision.fog.smithi130:Scheduling deploy of rhel 8.0 2020-01-16T00:45:09.565 INFO:teuthology.orchestra.console:Power off smithi086 2020-01-16T00:45:09.566 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-01-16T00:45:09.588 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:45:09.623 INFO:teuthology.orchestra.console:Power off smithi130 2020-01-16T00:45:09.624 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-01-16T00:45:09.649 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:45:09.738 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2020-01-16T00:45:09.771 INFO:teuthology.orchestra.console:Power off for smithi130 completed 2020-01-16T00:45:09.839 INFO:teuthology.orchestra.console:Power on smithi086 2020-01-16T00:45:09.840 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-01-16T00:45:09.863 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:45:09.872 INFO:teuthology.orchestra.console:Power on smithi130 2020-01-16T00:45:09.872 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-01-16T00:45:09.893 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:45:13.890 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:45:13.919 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:45:14.100 INFO:teuthology.orchestra.console:Power on for smithi086 completed 2020-01-16T00:45:14.201 INFO:teuthology.provision.fog.smithi086:Waiting for deploy to finish 2020-01-16T00:45:18.030 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:45:18.240 INFO:teuthology.orchestra.console:Power on for smithi130 completed 2020-01-16T00:45:18.341 INFO:teuthology.provision.fog.smithi130:Waiting for deploy to finish 2020-01-16T00:45:35.527 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2020-01-16T00:45:35.527 DEBUG:teuthology.task.console_log:Killing console logger for smithi086 2020-01-16T00:45:35.528 DEBUG:teuthology.task.console_log:Killing console logger for smithi130 2020-01-16T00:45:35.528 DEBUG:teuthology.exit:Finished running handlers