2020-11-19T21:45:07.138 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-19T21:45:07.140 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2020-11-19_17:21:50-rados-wip-yuri4-testing-2020-11-19-0746-nautilus-distro-basic-smithi --block --description rados/singleton/{all/thrash_cache_writeback_proxy_none msgr-failures/many msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_latest}} --lock --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2020-11-19_17:21:50-rados-wip-yuri4-testing-2020-11-19-0746-nautilus-distro-basic-smithi/5637917 -- /tmp/teuthology-worker.ymips_n0.tmp 2020-11-19T21:45:07.176 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-19T21:45:07.416 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2020-11-19_17:21:50-rados-wip-yuri4-testing-2020-11-19-0746-nautilus-distro-basic-smithi/5637917 branch: wip-yuri4-testing-2020-11-19-0746-nautilus description: rados/singleton/{all/thrash_cache_writeback_proxy_none msgr-failures/many msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_latest}} email: ceph-qa@ceph.io first_in_suite: false job_id: '5637917' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2020-11-19_17:21:50-rados-wip-yuri4-testing-2020-11-19-0746-nautilus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: rhel os_version: '7.8' overrides: admin_socket: branch: wip-yuri4-testing-2020-11-19-0746-nautilus ceph: conf: global: mon client hunt interval max multiple: 2 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd debug verify cached snaps: true osd debug verify missing on start: true osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd sloppy crc: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME mon_bind_msgr2: false sha1: a7ab790d3358b8b54c154754116c26711018b37e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: sha1: a7ab790d3358b8b54c154754116c26711018b37e workunit: branch: wip-yuri4-testing-2020-11-19-0746-nautilus sha1: a7ab790d3358b8b54c154754116c26711018b37e owner: scheduled_yuriw@teuthology priority: 80 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - - osd.3 - osd.4 - osd.5 - client.0 sha1: a7ab790d3358b8b54c154754116c26711018b37e sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri4-testing-2020-11-19-0746-nautilus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2020-11-19-0746-nautilus/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a7ab790d3358b8b54c154754116c26711018b37e tasks: - install: null - ceph: log-whitelist: - but it is still running - slow request - overall HEALTH_ - \(CACHE_POOL_ - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 500 - background_exec: mon.a: - while true - do sleep 30 - echo proxy - sudo ceph osd tier cache-mode cache proxy - sleep 10 - sudo ceph osd pool set cache cache_target_full_ratio .001 - echo cache-try-flush-evict-all - rados -p cache cache-try-flush-evict-all - sleep 5 - echo cache-flush-evict-all - rados -p cache cache-flush-evict-all - sleep 5 - echo remove overlay - sudo ceph osd tier remove-overlay base - sleep 20 - echo add writeback overlay - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd pool set cache cache_target_full_ratio .8 - sudo ceph osd tier set-overlay base cache - sleep 30 - sudo ceph osd tier cache-mode cache readproxy - done - rados: clients: - client.0 max_seconds: 600 objects: 10000 op_weights: copy_from: 50 delete: 50 read: 100 write: 100 ops: 400000 pools: - base size: 1024 teuthology_branch: master timestamp: 2020-11-19_17:21:50 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.6665 2020-11-19T21:45:07.417 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2020-11-19-0746-nautilus/qa; will attempt to use it 2020-11-19T21:45:07.417 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2020-11-19-0746-nautilus/qa/tasks 2020-11-19T21:45:07.418 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-19T21:45:07.426 INFO:teuthology.task.internal:Checking packages... 2020-11-19T21:45:07.459 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'a7ab790d3358b8b54c154754116c26711018b37e' 2020-11-19T21:45:07.460 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-19T21:45:07.460 INFO:teuthology.packaging:ref: None 2020-11-19T21:45:07.461 INFO:teuthology.packaging:tag: None 2020-11-19T21:45:07.461 INFO:teuthology.packaging:branch: wip-yuri4-testing-2020-11-19-0746-nautilus 2020-11-19T21:45:07.461 INFO:teuthology.packaging:sha1: a7ab790d3358b8b54c154754116c26711018b37e 2020-11-19T21:45:07.462 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-yuri4-testing-2020-11-19-0746-nautilus 2020-11-19T21:45:07.634 INFO:teuthology.task.internal:Found packages for ceph version 14.2.14-4.ga7ab790d 2020-11-19T21:45:07.634 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-19T21:45:07.644 INFO:teuthology.task.internal:no buildpackages task found 2020-11-19T21:45:07.645 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-19T21:45:07.675 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-19T21:45:07.676 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-19T21:45:07.915 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:45:17.934 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:45:27.950 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:45:37.973 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:45:47.999 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:45:58.023 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:46:08.047 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:46:18.063 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:46:28.080 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:46:38.101 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:46:48.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:46:58.132 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:47:08.147 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:47:18.166 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:47:28.183 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:47:38.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:47:48.217 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:47:58.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:48:08.261 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:48:18.284 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:48:28.302 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:48:38.316 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:48:48.330 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:48:58.366 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:49:08.375 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:49:18.404 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:49:28.412 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:49:38.442 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T21:49:48.458 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T21:49:58.475 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:50:08.485 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:50:18.502 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:50:28.531 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:50:38.546 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:50:48.567 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:50:58.590 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:51:08.605 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:51:18.926 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:51:28.934 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:51:38.965 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:51:48.986 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:51:59.014 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:52:09.041 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-19T21:52:19.058 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T21:52:29.082 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T21:52:39.103 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T21:52:49.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T21:52:59.142 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T21:53:09.160 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-19T21:53:19.171 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)...