2020-04-07T15:04:50.702 INFO:root:teuthology version: 1.0.0-b1b3d82 2020-04-07T15:04:50.704 DEBUG:teuthology.run:Teuthology command: teuthology --lock --archive /home/teuthworker/archive/kchai-2020-04-07_14:37:58-rados-wip-kefu-testing-2020-04-07-1603-distro-basic-smithi/4931151 --owner scheduled_kchai@teuthology -- --verbose --block --description rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/nautilus-v2only.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_7.6.yaml} msgr-failures/few.yaml rados.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} --name kchai-2020-04-07_14:37:58-rados-wip-kefu-testing-2020-04-07-1603-distro-basic-smithi /tmp/teuthology-worker.Jd5Mi1.tmp 2020-04-07T15:04:50.742 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-04-07T15:04:50.801 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-04-07_14:37:58-rados-wip-kefu-testing-2020-04-07-1603-distro-basic-smithi/4931151 branch: wip-kefu-testing-2020-04-07-1603 description: rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/nautilus-v2only.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_7.6.yaml} msgr-failures/few.yaml rados.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} email: null first_in_suite: false job_id: '4931151' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-04-07_14:37:58-rados-wip-kefu-testing-2020-04-07-1603-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-kefu-testing-2020-04-07-1603 ceph: conf: global: mon client directed command retry: 5 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 filestore odsync write: true osd backoff on degraded: true osd backoff on peering: true osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd max backfills: 2 osd max markdown count: 1000 osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 0.5 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: f66d07eeb60a8cf6e623ab16f545b8e195f4d964 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: f66d07eeb60a8cf6e623ab16f545b8e195f4d964 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-kefu-testing-2020-04-07-2234 sha1: 2a27f6eef9dc660ad72ab6b9373d474c164cdf6a owner: scheduled_kchai@teuthology priority: 30 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 sha1: f66d07eeb60a8cf6e623ab16f545b8e195f4d964 suite: rados suite_branch: wip-kefu-testing-2020-04-07-2234 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-kefu-testing-2020-04-07-2234/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 2a27f6eef9dc660ad72ab6b9373d474c164cdf6a tasks: - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: null - thrashosds: chance_pgnum_grow: 2 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.2 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: wip-py3 tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.14483 2020-04-07T15:04:50.802 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-kefu-testing-2020-04-07-2234/qa; will attempt to use it 2020-04-07T15:04:50.802 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-kefu-testing-2020-04-07-2234/qa/tasks 2020-04-07T15:04:50.803 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-04-07T15:04:50.805 INFO:teuthology.task.internal:Checking packages... 2020-04-07T15:04:50.822 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'f66d07eeb60a8cf6e623ab16f545b8e195f4d964' 2020-04-07T15:04:50.822 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-04-07T15:04:50.823 INFO:teuthology.packaging:ref: None 2020-04-07T15:04:50.823 INFO:teuthology.packaging:tag: None 2020-04-07T15:04:50.824 INFO:teuthology.packaging:branch: wip-kefu-testing-2020-04-07-1603 2020-04-07T15:04:50.824 INFO:teuthology.packaging:sha1: f66d07eeb60a8cf6e623ab16f545b8e195f4d964 2020-04-07T15:04:50.825 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-kefu-testing-2020-04-07-1603 2020-04-07T15:04:51.069 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-486.gf66d07e 2020-04-07T15:04:51.070 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-04-07T15:04:51.080 INFO:teuthology.task.internal:no buildpackages task found 2020-04-07T15:04:51.080 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-04-07T15:04:51.156 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-04-07T15:04:51.157 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-04-07T15:04:51.205 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:05:01.221 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:05:11.231 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:05:21.250 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:05:31.263 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:05:41.281 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:05:51.298 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:06:01.314 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:06:11.329 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:06:21.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:06:31.366 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:06:41.383 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:06:51.395 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:07:01.414 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:07:11.431 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:07:21.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:07:31.466 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:07:41.481 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:07:51.494 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:08:01.509 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:08:11.527 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:08:21.541 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:08:31.554 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:08:41.570 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:08:51.585 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:09:01.609 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:09:11.628 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:09:21.647 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:09:31.658 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:09:41.668 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:09:51.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:10:01.698 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:10:11.716 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:10:21.733 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:10:31.750 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:10:41.773 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:10:51.790 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:11:01.805 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:11:11.825 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-04-07T15:11:21.845 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:11:31.861 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:11:41.873 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:11:51.892 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:12:01.908 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:12:11.925 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:12:21.945 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:12:31.961 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:12:41.978 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:12:51.992 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-04-07T15:13:02.009 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:13:12.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:13:22.041 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:13:32.057 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:13:42.073 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-04-07T15:13:52.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...